2024-03-27T09:22:15.912 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T09:22:15.912 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} --name matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965 --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965/orig.config.yaml 2024-03-27T09:22:15.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T09:22:16.044 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965 branch: wip-sjust-crimson-testing-2024-03-20 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} email: null first_in_suite: false job_id: '7625965' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-sjust-crimson-testing-2024-03-20 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: ad2494b32de08d5bf12408a9a641b9ebfb123d09 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: ad2494b32de08d5bf12408a9a641b9ebfb123d09 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-matanb-crimson-thrash-osd env: CRIMSON_COMPAT: '1' sha1: 6db07181a4df6107bd2ae48adf3d2421c02fa2c2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 8843 sha1: ad2494b32de08d5bf12408a9a641b9ebfb123d09 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-thrash-osd suite_path: /home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa suite_relpath: qa suite_repo: https://github.com/Matan-B/ceph.git suite_sha1: 6db07181a4df6107bd2ae48adf3d2421c02fa2c2 targets: smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxkoIjMjUjtIe7Wmc5+7Wr2fHSgfxRSnOQ7Ih8nYqPiMa6fdfWXC9Hao0kOmzN3OmyR7icp1d20917tGueayzg= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbyDdzKHVWfMa4Qf1LA5apN8ds2mpDxV5Qmfe4sXxBRhk8MyiGkZjKsleV3XsqJ6tgYL4+MCbMbFDDS29cPE4c= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 localize_reads: true objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_08:49:03 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T09:22:16.044 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa; will attempt to use it 2024-03-27T09:22:16.045 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks 2024-03-27T09:22:16.045 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T09:22:16.046 INFO:teuthology.task.internal:Checking packages... 2024-03-27T09:22:16.070 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'ad2494b32de08d5bf12408a9a641b9ebfb123d09' 2024-03-27T09:22:16.070 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T09:22:16.070 INFO:teuthology.packaging:ref: None 2024-03-27T09:22:16.070 INFO:teuthology.packaging:tag: None 2024-03-27T09:22:16.071 INFO:teuthology.packaging:branch: wip-sjust-crimson-testing-2024-03-20 2024-03-27T09:22:16.071 INFO:teuthology.packaging:sha1: ad2494b32de08d5bf12408a9a641b9ebfb123d09 2024-03-27T09:22:16.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-sjust-crimson-testing-2024-03-20 2024-03-27T09:22:16.214 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2247.gad2494b3 2024-03-27T09:22:16.216 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T09:22:16.222 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T09:22:16.222 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T09:22:16.284 INFO:teuthology.task.internal:Saving configuration 2024-03-27T09:22:16.300 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T09:22:16.310 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T09:22:16.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 09:16:57.881828', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxkoIjMjUjtIe7Wmc5+7Wr2fHSgfxRSnOQ7Ih8nYqPiMa6fdfWXC9Hao0kOmzN3OmyR7icp1d20917tGueayzg='} 2024-03-27T09:22:16.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 09:16:57.880559', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbyDdzKHVWfMa4Qf1LA5apN8ds2mpDxV5Qmfe4sXxBRhk8MyiGkZjKsleV3XsqJ6tgYL4+MCbMbFDDS29cPE4c='} 2024-03-27T09:22:16.356 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T09:22:16.381 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-03-27T09:22:16.382 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-27T09:22:16.382 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T09:22:16.463 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f07baae8e50>, signals=[15]) 2024-03-27T09:22:16.463 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T09:22:16.470 INFO:teuthology.task.internal:Opening connections... 2024-03-27T09:22:16.470 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-27T09:22:16.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:22:16.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-27T09:22:16.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:22:16.655 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T09:22:16.662 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-27T09:22:16.680 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-27T09:22:16.680 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-27T09:22:16.737 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-27T09:22:16.737 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-27T09:22:16.737 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T09:22:16.738 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T09:22:16.739 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-27T09:22:16.767 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-27T09:22:16.785 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-27T09:22:16.786 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-27T09:22:16.846 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-27T09:22:16.846 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-27T09:22:16.846 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-27T09:22:16.846 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-27T09:22:16.846 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T09:22:16.847 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T09:22:16.847 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-27T09:22:16.867 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T09:22:16.876 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T09:22:16.990 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T09:22:16.990 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-27T09:22:16.993 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-27T09:22:17.009 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T09:22:17.015 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T09:22:17.015 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-27T09:22:17.049 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-27T09:22:17.160 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T09:22:17.216 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T09:22:17.233 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-27T09:22:17.233 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-27T09:22:17.233 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T09:22:17.234 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T09:22:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T09:22:17.251 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-27T09:22:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T09:22:17.252 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-27T09:22:17.310 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-27T09:22:17.311 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-27T09:22:17.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-27T09:22:17.312 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-03-27T09:22:17.312 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-03-27T09:22:17.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-03-27T09:22:19.316 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-27T09:22:19.440 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-27T09:22:20.916 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-27T09:22:21.049 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-27T09:22:21.882 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-27T09:22:25.383 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-27T09:22:29.825 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-27T09:22:30.338 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 919 kB/s | 8.0 MB 00:08 2024-03-27T09:22:31.201 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-27T09:22:32.922 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-27T09:22:34.160 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-27T09:22:37.598 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-27T09:22:39.082 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-27T09:22:39.110 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-27T09:22:40.519 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-27T09:22:41.275 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T09:22:41.276 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T09:22:41.413 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T09:22:41.421 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:22:41.422 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:22:41.422 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:22:41.422 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-27T09:22:41.422 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:22:41.423 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 76 M 2024-03-27T09:22:41.423 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-27T09:22:41.423 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:22:41.920 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 190 kB/s | 66 kB 00:00 2024-03-27T09:22:45.483 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-27T09:22:47.607 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T09:22:47.607 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T09:22:47.690 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T09:22:47.699 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 76 M 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-03-27T09:22:47.700 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:22:48.093 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 186 kB/s | 66 kB 00:00 2024-03-27T09:22:52.485 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.4 MB/s | 15 MB 00:10 2024-03-27T09:23:00.800 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.1 MB/s | 15 MB 00:13 2024-03-27T09:23:04.151 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.5 MB/s | 33 MB 00:22 2024-03-27T09:23:07.909 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.1 MB/s | 27 MB 00:25 2024-03-27T09:23:07.910 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:23:07.910 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.9 MB/s | 76 MB 00:26 2024-03-27T09:23:08.578 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:23:09.012 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:23:09.012 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:23:10.011 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:23:10.011 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:23:10.032 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.3 MB/s | 27 MB 00:21 2024-03-27T09:23:12.469 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:23:13.067 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T09:23:13.160 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:23:15.773 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.2 MB/s | 33 MB 00:28 2024-03-27T09:23:15.773 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:23:15.773 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.7 MB/s | 76 MB 00:28 2024-03-27T09:23:15.890 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:23:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:23:16.437 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:23:16.811 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:23:16.811 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:23:17.809 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:23:17.809 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:23:20.273 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:23:20.731 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:23:20.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:20.880 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T09:23:20.965 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:23:23.726 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:23:23.853 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:23:25.583 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:28.589 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:23:28.628 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:33.406 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:54.656 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:54.657 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T09:23:54.657 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:23:54.700 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:55.094 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:55.094 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T09:23:55.094 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:23:55.094 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:23:55.472 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:23:55.795 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T09:23:55.891 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.892 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T09:23:55.892 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T09:23:55.892 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T09:23:55.892 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T09:23:55.892 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-27T09:23:55.892 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T09:23:55.892 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T09:23:55.972 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T09:23:55.972 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T09:23:55.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T09:23:56.007 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:23:56.008 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T09:23:56.009 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T09:23:56.010 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-432.el9.x86_64 2024-03-27T09:23:56.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T09:23:56.604 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-27T09:23:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T09:23:58.057 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-27T09:23:58.060 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-27T09:23:58.080 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.qOfAv5ZwS7 2024-03-27T09:23:58.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qOfAv5ZwS7 2024-03-27T09:23:58.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.qOfAv5ZwS7 2024-03-27T09:23:58.303 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.qOfAv5ZwS7 is 6KB 2024-03-27T09:23:58.352 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.qOfAv5ZwS7 2024-03-27T09:23:58.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T09:23:58.431 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T09:23:58.431 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T09:23:58.431 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T09:23:58.431 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T09:23:58.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T09:23:58.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-27T09:24:02.784 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:24:02.785 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T09:24:02.785 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:24:02.831 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:24:03.223 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:24:03.224 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T09:24:03.224 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T09:24:03.224 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T09:24:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:24:03.653 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:24:03.984 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T09:24:04.092 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T09:24:04.093 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T09:24:04.093 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T09:24:04.093 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T09:24:04.093 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T09:24:04.093 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-03-27T09:24:04.093 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T09:24:04.093 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T09:24:04.174 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T09:24:04.174 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T09:24:04.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-03-27T09:24:04.209 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-03-27T09:24:04.209 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-03-27T09:24:04.209 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-03-27T09:24:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:24:04.211 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T09:24:04.212 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T09:24:04.212 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-432.el9.x86_64 2024-03-27T09:24:04.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T09:24:04.797 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-27T09:24:06.134 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T09:24:06.173 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-27T09:24:06.175 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-27T09:24:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.aXDa00J9Gy 2024-03-27T09:24:06.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aXDa00J9Gy 2024-03-27T09:24:06.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.aXDa00J9Gy 2024-03-27T09:24:06.400 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.aXDa00J9Gy is 6KB 2024-03-27T09:24:06.447 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.aXDa00J9Gy 2024-03-27T09:24:06.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T09:24:06.526 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T09:24:06.526 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T09:24:06.526 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T09:24:06.526 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T09:24:06.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T09:24:06.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-27T09:24:28.618 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-27T09:24:28.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:24:28.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:24:36.678 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-03-27T09:24:36.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:24:36.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:24:47.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-27T09:24:55.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-27T09:24:56.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:24:56.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:25:04.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:25:04.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:25:07.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-27T09:25:19.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:25:19.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:25:56.035 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T09:26:08.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:26:08.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:26:08.410 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-27T09:26:08.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:26:08.800 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-27T09:26:08.800 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-27T09:26:08.815 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-432.el9.x86_64 2024-03-27T09:26:08.816 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T09:26:08.816 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T09:26:08.816 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T09:26:09.817 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-27T09:26:09.817 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T09:26:09.892 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-27T09:26:09.932 DEBUG:teuthology.parallel:result is None 2024-03-27T09:26:19.276 DEBUG:teuthology.orchestra.remote:timed out 2024-03-27T09:26:34.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:26:34.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:26:34.725 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-27T09:26:35.130 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:26:35.131 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-432.el9.x86_64"... 2024-03-27T09:26:35.131 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-27T09:26:35.146 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-432.el9.x86_64 2024-03-27T09:26:35.146 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T09:26:35.146 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T09:26:35.146 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T09:26:36.147 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-03-27T09:26:36.148 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T09:26:36.207 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-27T09:26:36.244 DEBUG:teuthology.parallel:result is None 2024-03-27T09:26:36.244 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T09:26:36.251 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T09:26:36.252 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T09:26:36.254 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T09:26:36.271 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T09:26:36.287 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T09:26:36.337 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T09:26:36.338 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T09:26:36.340 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T09:26:36.365 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T09:26:36.387 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T09:26:36.387 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T09:26:36.402 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T09:26:36.441 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T09:26:36.443 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T09:26:36.456 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T09:26:36.457 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T09:26:36.459 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T09:26:36.466 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T09:26:36.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T09:26:36.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T09:26:36.530 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T09:26:36.539 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T09:26:36.539 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T09:26:36.571 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T09:26:36.587 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T09:26:36.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T09:26:36.724 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T09:26:36.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T09:26:36.854 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:26:36.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T09:26:36.922 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T09:26:36.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T09:26:37.018 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T09:26:37.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T09:26:37.144 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:26:37.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T09:26:37.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-27T09:26:37.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-27T09:26:37.259 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T09:26:37.294 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T09:26:37.741 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T09:26:37.752 INFO:teuthology.task.internal:Starting timer... 2024-03-27T09:26:37.752 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T09:26:37.800 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T09:26:37.848 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-27T09:26:37.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-27T09:26:37.914 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-27T09:26:37.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-27T09:26:37.967 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-27T09:26:37.969 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T09:26:37.969 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-27T09:26:37.998 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-27T09:26:37.999 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-27T09:26:38.024 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-27T09:26:38.025 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi170.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-03-27T09:26:38.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T09:26:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T09:26:38.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T09:26:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T09:26:38.094 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T09:26:38.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-27T09:26:38.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-27T09:26:38.176 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T09:26:38.187 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T09:26:38.205 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T09:26:38.207 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi170.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T09:30:54.368 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi170.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-03-27T09:30:54.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:30:54.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:30:54.461 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-27T09:30:54.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-27T09:30:54.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:30:54.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T09:30:54.631 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-27T09:30:54.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-27T09:30:54.714 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T09:30:54.725 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T09:30:54.725 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T09:30:54.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T09:30:54.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T09:30:54.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T09:30:54.757 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T09:30:54.775 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T09:30:54.785 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T09:30:54.802 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T09:30:54.805 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-27T09:30:54.818 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-27T09:30:54.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-27T09:30:54.835 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T09:30:54.845 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-27T09:30:54.851 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T09:30:54.862 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T09:30:54.878 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T09:30:54.899 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-27T09:30:54.902 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T09:30:54.902 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-27T09:30:54.903 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.903 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.903 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.903 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.935 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T09:30:54.939 INFO:teuthology.run_tasks:Running task install... 2024-03-27T09:30:55.033 DEBUG:teuthology.task.install:project ceph 2024-03-27T09:30:55.034 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}} 2024-03-27T09:30:55.034 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'} 2024-03-27T09:30:55.034 INFO:teuthology.task.install:Using flavor: crimson 2024-03-27T09:30:55.039 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T09:30:55.040 INFO:teuthology.task.install:extra packages: [] 2024-03-27T09:30:55.040 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09', 'tag': None, 'wait_for_package': False} 2024-03-27T09:30:55.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ad2494b32de08d5bf12408a9a641b9ebfb123d09 2024-03-27T09:30:55.042 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09', 'tag': None, 'wait_for_package': False} 2024-03-27T09:30:55.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ad2494b32de08d5bf12408a9a641b9ebfb123d09 2024-03-27T09:30:55.182 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/ 2024-03-27T09:30:55.182 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2247.gad2494b3 2024-03-27T09:30:55.195 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/ 2024-03-27T09:30:55.195 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2247.gad2494b3 2024-03-27T09:30:55.433 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T09:30:55.433 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:30:55.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T09:30:55.436 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sjust-crimson-testing-2024-03-20/ad2494b32de08d5bf12408a9a641b9ebfb123d09/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T09:30:55.436 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:30:55.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T09:30:55.464 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T09:30:55.464 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ad2494b32de08d5bf12408a9a641b9ebfb123d09/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T09:30:55.466 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T09:30:55.466 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ad2494b32de08d5bf12408a9a641b9ebfb123d09/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T09:30:55.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T09:30:55.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T09:30:55.614 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T09:30:55.615 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T09:30:55.681 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-03-27T09:30:55.682 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-27T09:30:55.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-03-27T09:30:55.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-27T09:30:55.919 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-27T09:30:55.924 INFO:teuthology.orchestra.run.smithi172.stdout:74 files removed 2024-03-27T09:30:55.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2024-03-27T09:30:55.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2024-03-27T09:30:56.815 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 154 kB/s | 87 kB 00:00 2024-03-27T09:30:56.842 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 145 kB/s | 87 kB 00:00 2024-03-27T09:30:57.232 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-27T09:30:57.384 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-27T09:30:57.668 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 5.1 kB/s | 2.0 kB 00:00 2024-03-27T09:30:58.068 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 3.2 kB/s | 2.0 kB 00:00 2024-03-27T09:30:58.252 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-27T09:30:58.606 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-27T09:30:59.952 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-27T09:31:00.306 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-27T09:31:01.167 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-03-27T09:31:02.030 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-27T09:31:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-27T09:31:04.623 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-27T09:31:08.734 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-27T09:31:10.424 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-27T09:31:11.234 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-27T09:31:13.943 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 815 kB/s | 6.1 MB 00:07 2024-03-27T09:31:15.452 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 16 kB 00:00 2024-03-27T09:31:16.159 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-27T09:31:16.923 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-27T09:31:18.645 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:31:18.649 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:18.649 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:31:18.649 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:18.649 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 9.6 M 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.4 M 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.7 M 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T09:31:18.650 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.9 M 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 19 M 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 25 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 795 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T09:31:18.651 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 548 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.0 M 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 45 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 131 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 160 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 315 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 293 k 2024-03-27T09:31:18.652 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 100 k 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout:Install 30 Packages 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2024-03-27T09:31:18.653 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:18.654 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 58 M 2024-03-27T09:31:18.654 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:31:21.710 INFO:teuthology.orchestra.run.smithi170.stdout:(1/32): ceph-radosgw-19.0.0-2247.gad2494b3.el9. 3.7 MB/s | 9.6 MB 00:02 2024-03-27T09:31:21.768 INFO:teuthology.orchestra.run.smithi170.stdout:(2/32): ceph-selinux-19.0.0-2247.gad2494b3.el9. 434 kB/s | 25 kB 00:00 2024-03-27T09:31:21.837 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-27T09:31:21.977 INFO:teuthology.orchestra.run.smithi170.stdout:(3/32): ceph-base-19.0.0-2247.gad2494b3.el9.x86 2.1 MB/s | 5.9 MB 00:02 2024-03-27T09:31:22.010 INFO:teuthology.orchestra.run.smithi170.stdout:(4/32): libcephfs2-19.0.0-2247.gad2494b3.el9.x8 3.2 MB/s | 795 kB 00:00 2024-03-27T09:31:22.194 INFO:teuthology.orchestra.run.smithi170.stdout:(5/32): libradosstriper1-19.0.0-2247.gad2494b3. 2.5 MB/s | 548 kB 00:00 2024-03-27T09:31:22.294 INFO:teuthology.orchestra.run.smithi170.stdout:(6/32): python3-ceph-argparse-19.0.0-2247.gad24 451 kB/s | 45 kB 00:00 2024-03-27T09:31:22.403 INFO:teuthology.orchestra.run.smithi170.stdout:(7/32): python3-ceph-common-19.0.0-2247.gad2494 1.2 MB/s | 131 kB 00:00 2024-03-27T09:31:22.870 INFO:teuthology.orchestra.run.smithi170.stdout:(8/32): librgw2-19.0.0-2247.gad2494b3.el9.x86_6 5.9 MB/s | 5.0 MB 00:00 2024-03-27T09:31:22.887 INFO:teuthology.orchestra.run.smithi170.stdout:(9/32): python3-cephfs-19.0.0-2247.gad2494b3.el 331 kB/s | 160 kB 00:00 2024-03-27T09:31:22.937 INFO:teuthology.orchestra.run.smithi170.stdout:(10/32): python3-rados-19.0.0-2247.gad2494b3.el 4.6 MB/s | 315 kB 00:00 2024-03-27T09:31:23.004 INFO:teuthology.orchestra.run.smithi170.stdout:(11/32): python3-rgw-19.0.0-2247.gad2494b3.el9. 1.5 MB/s | 100 kB 00:00 2024-03-27T09:31:23.029 INFO:teuthology.orchestra.run.smithi170.stdout:(12/32): python3-rbd-19.0.0-2247.gad2494b3.el9. 2.0 MB/s | 293 kB 00:00 2024-03-27T09:31:23.146 INFO:teuthology.orchestra.run.smithi170.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-03-27T09:31:23.172 INFO:teuthology.orchestra.run.smithi170.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-27T09:31:23.196 INFO:teuthology.orchestra.run.smithi170.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-27T09:31:23.280 INFO:teuthology.orchestra.run.smithi170.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-27T09:31:23.306 INFO:teuthology.orchestra.run.smithi170.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-27T09:31:23.322 INFO:teuthology.orchestra.run.smithi170.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-27T09:31:23.398 INFO:teuthology.orchestra.run.smithi170.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 478 kB/s | 36 kB 00:00 2024-03-27T09:31:23.465 INFO:teuthology.orchestra.run.smithi170.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2024-03-27T09:31:23.490 INFO:teuthology.orchestra.run.smithi170.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-03-27T09:31:24.007 INFO:teuthology.orchestra.run.smithi170.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-27T09:31:24.266 INFO:teuthology.orchestra.run.smithi170.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-27T09:31:24.272 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 9.6 M 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.4 M 2024-03-27T09:31:24.283 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.7 M 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.9 M 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 19 M 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 25 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 795 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T09:31:24.284 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 548 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.0 M 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 45 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 131 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 160 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 315 k 2024-03-27T09:31:24.285 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 293 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 100 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout:Install 30 Packages 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2024-03-27T09:31:24.286 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:24.287 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 58 M 2024-03-27T09:31:24.287 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:31:24.441 INFO:teuthology.orchestra.run.smithi170.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-27T09:31:24.608 INFO:teuthology.orchestra.run.smithi170.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-27T09:31:24.783 INFO:teuthology.orchestra.run.smithi170.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-27T09:31:25.034 INFO:teuthology.orchestra.run.smithi170.stdout:(27/32): ceph-common-19.0.0-2247.gad2494b3.el9. 3.2 MB/s | 19 MB 00:05 2024-03-27T09:31:25.376 INFO:teuthology.orchestra.run.smithi170.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 324 kB/s | 192 kB 00:00 2024-03-27T09:31:25.459 INFO:teuthology.orchestra.run.smithi170.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 985 kB/s | 838 kB 00:00 2024-03-27T09:31:25.784 INFO:teuthology.orchestra.run.smithi170.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-27T09:31:26.768 INFO:teuthology.orchestra.run.smithi170.stdout:(31/32): librbd1-19.0.0-2247.gad2494b3.el9.x86_ 2.8 MB/s | 3.7 MB 00:01 2024-03-27T09:31:26.826 INFO:teuthology.orchestra.run.smithi170.stdout:(32/32): librados2-19.0.0-2247.gad2494b3.el9.x8 2.3 MB/s | 3.4 MB 00:01 2024-03-27T09:31:26.831 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:31:26.831 INFO:teuthology.orchestra.run.smithi170.stdout:Total 7.1 MB/s | 58 MB 00:08 2024-03-27T09:31:26.866 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:31:26.904 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:31:26.904 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:31:27.378 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:31:27.378 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:31:28.279 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-radosgw-19.0.0-2247.gad2494b3.el9. 2.7 MB/s | 9.6 MB 00:03 2024-03-27T09:31:28.338 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-selinux-19.0.0-2247.gad2494b3.el9. 432 kB/s | 25 kB 00:00 2024-03-27T09:31:28.563 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): libcephfs2-19.0.0-2247.gad2494b3.el9.x8 3.5 MB/s | 795 kB 00:00 2024-03-27T09:31:28.681 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): ceph-base-19.0.0-2247.gad2494b3.el9.x86 1.5 MB/s | 5.9 MB 00:03 2024-03-27T09:31:28.772 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): libradosstriper1-19.0.0-2247.gad2494b3. 2.6 MB/s | 548 kB 00:00 2024-03-27T09:31:28.815 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:31:28.839 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): python3-ceph-argparse-19.0.0-2247.gad24 678 kB/s | 45 kB 00:00 2024-03-27T09:31:28.914 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-ceph-common-19.0.0-2247.gad2494 1.7 MB/s | 131 kB 00:00 2024-03-27T09:31:28.989 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): python3-cephfs-19.0.0-2247.gad2494b3.el 2.1 MB/s | 160 kB 00:00 2024-03-27T09:31:29.115 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): python3-rados-19.0.0-2247.gad2494b3.el9 2.5 MB/s | 315 kB 00:00 2024-03-27T09:31:29.240 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): python3-rbd-19.0.0-2247.gad2494b3.el9. 2.3 MB/s | 293 kB 00:00 2024-03-27T09:31:29.315 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-rgw-19.0.0-2247.gad2494b3.el9. 1.3 MB/s | 100 kB 00:00 2024-03-27T09:31:29.320 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T09:31:29.353 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:29.424 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-27T09:31:29.453 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:29.466 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-27T09:31:29.491 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-27T09:31:29.541 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 3/34 2024-03-27T09:31:29.541 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-27T09:31:29.571 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:29.583 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-27T09:31:29.608 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-27T09:31:29.634 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:29.677 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-27T09:31:29.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-27T09:31:29.734 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-27T09:31:29.766 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T09:31:29.782 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T09:31:29.859 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 883 kB/s | 221 kB 00:00 2024-03-27T09:31:29.918 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-03-27T09:31:29.925 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T09:31:29.957 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T09:31:29.976 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-03-27T09:31:30.143 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-03-27T09:31:30.307 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-27T09:31:30.336 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:30.403 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:30.437 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9. 12/34 2024-03-27T09:31:30.492 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 13/34 2024-03-27T09:31:30.525 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 14/34 2024-03-27T09:31:30.603 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-27T09:31:30.647 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 16/34 2024-03-27T09:31:30.677 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): ceph-common-19.0.0-2247.gad2494b3.el9. 3.2 MB/s | 19 MB 00:05 2024-03-27T09:31:30.728 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 16/34 2024-03-27T09:31:30.760 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-27T09:31:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-27T09:31:30.835 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 650 kB/s | 49 kB 00:00 2024-03-27T09:31:30.882 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-27T09:31:30.923 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-27T09:31:30.936 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-27T09:31:30.963 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-27T09:31:30.995 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-27T09:31:31.003 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-03-27T09:31:31.024 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-27T09:31:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-27T09:31:31.086 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): librgw2-19.0.0-2247.gad2494b3.el9.x86_ 2.1 MB/s | 5.0 MB 00:02 2024-03-27T09:31:31.095 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T09:31:31.112 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-27T09:31:31.212 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-27T09:31:31.479 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-03-27T09:31:31.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T09:31:31.661 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 26/34 2024-03-27T09:31:31.716 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 26/34 2024-03-27T09:31:31.785 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 27/34 2024-03-27T09:31:31.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x8 28/34 2024-03-27T09:31:32.004 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): librbd1-19.0.0-2247.gad2494b3.el9.x86_ 4.6 MB/s | 3.7 MB 00:00 2024-03-27T09:31:32.421 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): librados2-19.0.0-2247.gad2494b3.el9.x8 2.6 MB/s | 3.4 MB 00:01 2024-03-27T09:31:32.424 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:31:32.424 INFO:teuthology.orchestra.run.smithi172.stdout:Total 7.1 MB/s | 58 MB 00:08 2024-03-27T09:31:32.457 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:31:32.497 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:31:32.497 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:31:32.975 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:31:32.975 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:31:34.205 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:34.252 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:31:34.926 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T09:31:34.964 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:34.998 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:35.039 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 30/34 2024-03-27T09:31:35.055 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:35.120 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 30/34 2024-03-27T09:31:35.120 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T09:31:35.120 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T09:31:35.120 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:35.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:35.152 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 3/34 2024-03-27T09:31:35.182 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:35.229 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:35.268 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-27T09:31:35.295 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-27T09:31:35.321 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-27T09:31:35.344 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T09:31:35.359 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T09:31:35.519 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T09:31:35.551 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T09:31:35.919 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-27T09:31:35.947 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:35.998 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:36.041 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9. 12/34 2024-03-27T09:31:36.097 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 13/34 2024-03-27T09:31:36.128 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 14/34 2024-03-27T09:31:36.215 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-27T09:31:36.251 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 16/34 2024-03-27T09:31:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 16/34 2024-03-27T09:31:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-27T09:31:36.483 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-27T09:31:36.517 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-27T09:31:36.557 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-27T09:31:36.590 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-27T09:31:36.618 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-27T09:31:36.654 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-27T09:31:36.700 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T09:31:37.232 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T09:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 26/34 2024-03-27T09:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 26/34 2024-03-27T09:31:37.387 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 27/34 2024-03-27T09:31:37.417 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x8 28/34 2024-03-27T09:31:39.763 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:39.805 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:40.492 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 29/34 2024-03-27T09:31:40.525 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 30/34 2024-03-27T09:31:40.631 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 30/34 2024-03-27T09:31:40.631 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T09:31:40.631 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T09:31:40.631 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:40.676 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-27T09:31:45.164 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:46.178 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 32/34 2024-03-27T09:31:46.203 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 32/34 2024-03-27T09:31:46.203 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:31:46.204 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T09:31:46.204 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T09:31:46.204 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T09:31:46.204 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:46.206 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T09:31:46.281 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T09:31:46.282 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:47.441 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 1/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 3/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 5/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 6/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 7/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9. 8/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x8 9/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 10/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 12/34 2024-03-27T09:31:47.442 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 13/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T09:31:47.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T09:31:47.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T09:31:47.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T09:31:47.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T09:31:47.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:47.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T09:31:47.445 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 33/34 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.490 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T09:31:48.491 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:48.492 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:31:48.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2024-03-27T09:31:49.333 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:33 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:31:49.485 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:31:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:49.486 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:31:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:49.486 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:31:49.486 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 87 M 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 173 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-27T09:31:49.487 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:31:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:31:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:Install 11 Packages 2024-03-27T09:31:49.488 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:31:49.488 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 90 M 2024-03-27T09:31:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 819 M 2024-03-27T09:31:49.489 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:31:50.095 INFO:teuthology.orchestra.run.smithi170.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-27T09:31:50.195 INFO:teuthology.orchestra.run.smithi170.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-27T09:31:50.236 INFO:teuthology.orchestra.run.smithi170.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-03-27T09:31:50.262 INFO:teuthology.orchestra.run.smithi170.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-27T09:31:50.287 INFO:teuthology.orchestra.run.smithi170.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.5 MB/s | 90 kB 00:00 2024-03-27T09:31:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:50.364 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-03-27T09:31:50.364 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-03-27T09:31:50.364 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-03-27T09:31:50.364 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-03-27T09:31:50.365 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-03-27T09:31:50.365 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-03-27T09:31:50.365 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-03-27T09:31:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:50.412 INFO:teuthology.orchestra.run.smithi170.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-27T09:31:50.621 INFO:teuthology.orchestra.run.smithi170.stdout:(7/11): libcephsqlite-19.0.0-2247.gad2494b3.el9 262 kB/s | 173 kB 00:00 2024-03-27T09:31:51.421 INFO:teuthology.orchestra.run.smithi170.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 233 kB/s | 187 kB 00:00 2024-03-27T09:31:51.523 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 32/34 2024-03-27T09:31:51.547 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 32/34 2024-03-27T09:31:51.548 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:31:51.548 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T09:31:51.548 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T09:31:51.548 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T09:31:51.548 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:51.550 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T09:31:51.630 INFO:teuthology.orchestra.run.smithi170.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 385 kB/s | 469 kB 00:01 2024-03-27T09:31:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T09:31:51.633 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:51.846 INFO:teuthology.orchestra.run.smithi170.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 287 kB/s | 122 kB 00:00 2024-03-27T09:31:52.827 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:52.827 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 1/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 3/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 4/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 5/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_6 6/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 7/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9. 8/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x8 9/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 10/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 11/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 12/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 13/34 2024-03-27T09:31:52.828 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T09:31:52.829 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T09:31:52.830 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T09:31:52.830 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T09:31:52.830 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 31/34 2024-03-27T09:31:52.831 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T09:31:52.831 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 33/34 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.994 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.995 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:53.996 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:31:54.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2024-03-27T09:31:54.919 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:33 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:31:55.068 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 87 M 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-27T09:31:55.069 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 173 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2024-03-27T09:31:55.070 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:31:55.071 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 90 M 2024-03-27T09:31:55.071 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 819 M 2024-03-27T09:31:55.071 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:31:55.765 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-27T09:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-27T09:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-27T09:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-27T09:31:56.033 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-03-27T09:31:56.116 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): libcephsqlite-19.0.0-2247.gad2494b3.el9 334 kB/s | 173 kB 00:00 2024-03-27T09:31:56.233 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-27T09:31:56.292 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-27T09:31:56.375 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.8 MB/s | 469 kB 00:00 2024-03-27T09:31:56.842 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 221 kB/s | 122 kB 00:00 2024-03-27T09:32:02.242 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): ceph-test-19.0.0-2247.gad2494b3.el9.x8 13 MB/s | 87 MB 00:06 2024-03-27T09:32:02.246 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:02.246 INFO:teuthology.orchestra.run.smithi172.stdout:Total 13 MB/s | 90 MB 00:07 2024-03-27T09:32:02.287 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:32:02.310 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:32:02.310 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:32:02.945 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:32:02.945 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:32:03.948 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:32:04.012 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-27T09:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-27T09:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T09:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T09:32:04.176 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-27T09:32:04.220 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-27T09:32:04.265 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-27T09:32:04.296 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-27T09:32:04.346 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-27T09:32:04.389 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-27T09:32:04.415 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 10/11 2024-03-27T09:32:13.795 INFO:teuthology.orchestra.run.smithi170.stdout:(11/11): ceph-test-19.0.0-2247.gad2494b3.el9.x8 3.7 MB/s | 87 MB 00:23 2024-03-27T09:32:13.798 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:13.798 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.7 MB/s | 90 MB 00:24 2024-03-27T09:32:13.839 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:32:13.862 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:32:13.863 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:32:14.510 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:32:14.511 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:32:15.514 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:32:15.585 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-27T09:32:15.620 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-27T09:32:15.646 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T09:32:15.726 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T09:32:15.771 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-27T09:32:15.823 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-27T09:32:15.877 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-27T09:32:15.901 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 10/11 2024-03-27T09:32:15.908 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-27T09:32:15.952 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-27T09:32:15.979 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 11/11 2024-03-27T09:32:15.992 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-27T09:32:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 10/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 11/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 1/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 2/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-27T09:32:22.149 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-16.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-27T09:32:22.571 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:22.572 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:32:22.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2024-03-27T09:32:23.470 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:02 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:32:23.623 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:32:23.625 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-27T09:32:23.625 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 6.5 k 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 2.2 M 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 1.6 M 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 248 k 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.0 M 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 71 M 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T09:32:23.626 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T09:32:23.627 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:32:23.628 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================= 2024-03-27T09:32:23.629 INFO:teuthology.orchestra.run.smithi172.stdout:Install 36 Packages 2024-03-27T09:32:23.629 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:23.629 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 84 M 2024-03-27T09:32:23.629 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 921 M 2024-03-27T09:32:23.629 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:32:24.039 INFO:teuthology.orchestra.run.smithi172.stdout:(1/36): ceph-19.0.0-2247.gad2494b3.el9.x86_64.r 36 kB/s | 6.5 kB 00:00 2024-03-27T09:32:24.830 INFO:teuthology.orchestra.run.smithi172.stdout:(2/36): ceph-mgr-19.0.0-2247.gad2494b3.el9.x86_ 1.6 MB/s | 1.6 MB 00:00 2024-03-27T09:32:25.064 INFO:teuthology.orchestra.run.smithi172.stdout:(3/36): ceph-mds-19.0.0-2247.gad2494b3.el9.x86_ 1.8 MB/s | 2.2 MB 00:01 2024-03-27T09:32:25.172 INFO:teuthology.orchestra.run.smithi172.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2247.gad24 2.2 MB/s | 248 kB 00:00 2024-03-27T09:32:25.289 INFO:teuthology.orchestra.run.smithi172.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-27T09:32:25.331 INFO:teuthology.orchestra.run.smithi172.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-27T09:32:25.365 INFO:teuthology.orchestra.run.smithi172.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-27T09:32:25.398 INFO:teuthology.orchestra.run.smithi172.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-27T09:32:25.432 INFO:teuthology.orchestra.run.smithi172.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-27T09:32:25.473 INFO:teuthology.orchestra.run.smithi172.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-27T09:32:25.540 INFO:teuthology.orchestra.run.smithi172.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-27T09:32:25.574 INFO:teuthology.orchestra.run.smithi172.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-27T09:32:25.608 INFO:teuthology.orchestra.run.smithi172.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-27T09:32:25.649 INFO:teuthology.orchestra.run.smithi172.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-27T09:32:25.767 INFO:teuthology.orchestra.run.smithi172.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-27T09:32:25.850 INFO:teuthology.orchestra.run.smithi172.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-27T09:32:26.234 INFO:teuthology.orchestra.run.smithi172.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-27T09:32:26.351 INFO:teuthology.orchestra.run.smithi172.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-27T09:32:26.409 INFO:teuthology.orchestra.run.smithi172.stdout:(19/36): ceph-mon-19.0.0-2247.gad2494b3.el9.x86 2.1 MB/s | 5.0 MB 00:02 2024-03-27T09:32:26.502 INFO:teuthology.orchestra.run.smithi172.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-27T09:32:26.643 INFO:teuthology.orchestra.run.smithi172.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-27T09:32:26.660 INFO:teuthology.orchestra.run.smithi172.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2024-03-27T09:32:26.719 INFO:teuthology.orchestra.run.smithi172.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-27T09:32:26.744 INFO:teuthology.orchestra.run.smithi172.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-03-27T09:32:26.803 INFO:teuthology.orchestra.run.smithi172.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-27T09:32:26.820 INFO:teuthology.orchestra.run.smithi172.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-27T09:32:26.887 INFO:teuthology.orchestra.run.smithi172.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-03-27T09:32:26.904 INFO:teuthology.orchestra.run.smithi172.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-03-27T09:32:27.079 INFO:teuthology.orchestra.run.smithi172.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-27T09:32:27.096 INFO:teuthology.orchestra.run.smithi172.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 85 kB/s | 16 kB 00:00 2024-03-27T09:32:27.188 INFO:teuthology.orchestra.run.smithi172.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 830 kB/s | 90 kB 00:00 2024-03-27T09:32:27.205 INFO:teuthology.orchestra.run.smithi172.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-03-27T09:32:27.322 INFO:teuthology.orchestra.run.smithi172.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-27T09:32:27.480 INFO:teuthology.orchestra.run.smithi172.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 126 kB/s | 20 kB 00:00 2024-03-27T09:32:27.506 INFO:teuthology.orchestra.run.smithi172.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-03-27T09:32:27.528 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 10/11 2024-03-27T09:32:27.600 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 11/11 2024-03-27T09:32:30.589 INFO:teuthology.orchestra.run.smithi172.stdout:(36/36): ceph-osd-19.0.0-2247.gad2494b3.el9.x86 12 MB/s | 71 MB 00:05 2024-03-27T09:32:30.594 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:30.594 INFO:teuthology.orchestra.run.smithi172.stdout:Total 12 MB/s | 84 MB 00:06 2024-03-27T09:32:30.619 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:32:30.663 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:32:30.663 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:32:31.390 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:32:31.390 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:32:32.975 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:32:33.041 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T09:32:33.074 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T09:32:33.102 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T09:32:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T09:32:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T09:32:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T09:32:33.333 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T09:32:33.403 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T09:32:33.444 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T09:32:33.508 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T09:32:33.562 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T09:32:33.600 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T09:32:33.649 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T09:32:33.704 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T09:32:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T09:32:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T09:32:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T09:32:34.011 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 11/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 1/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 2/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-27T09:32:34.026 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-27T09:32:34.027 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-27T09:32:34.041 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T09:32:34.117 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T09:32:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T09:32:34.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T09:32:34.225 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T09:32:34.260 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T09:32:34.328 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:34.407 INFO:teuthology.orchestra.run.smithi170.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:34.408 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:32:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T09:32:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T09:32:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T09:32:34.666 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T09:32:34.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2024-03-27T09:32:34.783 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T09:32:34.969 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9. 31/36 2024-03-27T09:32:34.997 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 32/36 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 32/36 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T09:32:35.023 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:35.285 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:19 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:32:35.436 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:32:35.438 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-27T09:32:35.438 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:32:35.438 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 6.5 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 2.2 M 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 1.6 M 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 248 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 5.0 M 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 71 M 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T09:32:35.439 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T09:32:35.440 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout:Install 36 Packages 2024-03-27T09:32:35.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:35.442 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 84 M 2024-03-27T09:32:35.442 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 921 M 2024-03-27T09:32:35.442 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:32:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:(1/36): ceph-19.0.0-2247.gad2494b3.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2024-03-27T09:32:36.262 INFO:teuthology.orchestra.run.smithi170.stdout:(2/36): ceph-mon-19.0.0-2247.gad2494b3.el9.x86_ 12 MB/s | 5.0 MB 00:00 2024-03-27T09:32:36.346 INFO:teuthology.orchestra.run.smithi170.stdout:(3/36): ceph-mds-19.0.0-2247.gad2494b3.el9.x86_ 3.3 MB/s | 2.2 MB 00:00 2024-03-27T09:32:36.446 INFO:teuthology.orchestra.run.smithi170.stdout:(4/36): ceph-mgr-19.0.0-2247.gad2494b3.el9.x86_ 2.1 MB/s | 1.6 MB 00:00 2024-03-27T09:32:36.471 INFO:teuthology.orchestra.run.smithi170.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2247.gad24 1.9 MB/s | 248 kB 00:00 2024-03-27T09:32:36.596 INFO:teuthology.orchestra.run.smithi170.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-27T09:32:36.621 INFO:teuthology.orchestra.run.smithi170.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-27T09:32:36.647 INFO:teuthology.orchestra.run.smithi170.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-27T09:32:36.672 INFO:teuthology.orchestra.run.smithi170.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-27T09:32:36.697 INFO:teuthology.orchestra.run.smithi170.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 765 kB/s | 39 kB 00:00 2024-03-27T09:32:36.723 INFO:teuthology.orchestra.run.smithi170.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.6 MB/s | 241 kB 00:00 2024-03-27T09:32:36.782 INFO:teuthology.orchestra.run.smithi170.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-27T09:32:36.807 INFO:teuthology.orchestra.run.smithi170.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-27T09:32:36.841 INFO:teuthology.orchestra.run.smithi170.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-27T09:32:36.866 INFO:teuthology.orchestra.run.smithi170.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-27T09:32:37.050 INFO:teuthology.orchestra.run.smithi170.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 600 kB/s | 125 kB 00:00 2024-03-27T09:32:37.241 INFO:teuthology.orchestra.run.smithi170.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 59 kB/s | 22 kB 00:00 2024-03-27T09:32:37.267 INFO:teuthology.orchestra.run.smithi170.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-27T09:32:37.367 INFO:teuthology.orchestra.run.smithi170.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-27T09:32:37.384 INFO:teuthology.orchestra.run.smithi170.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 305 kB/s | 43 kB 00:00 2024-03-27T09:32:37.434 INFO:teuthology.orchestra.run.smithi170.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-27T09:32:37.459 INFO:teuthology.orchestra.run.smithi170.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-27T09:32:37.485 INFO:teuthology.orchestra.run.smithi170.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-27T09:32:37.510 INFO:teuthology.orchestra.run.smithi170.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-03-27T09:32:37.535 INFO:teuthology.orchestra.run.smithi170.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-27T09:32:37.552 INFO:teuthology.orchestra.run.smithi170.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 470 kB/s | 20 kB 00:00 2024-03-27T09:32:37.594 INFO:teuthology.orchestra.run.smithi170.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 787 kB/s | 46 kB 00:00 2024-03-27T09:32:37.628 INFO:teuthology.orchestra.run.smithi170.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-27T09:32:37.670 INFO:teuthology.orchestra.run.smithi170.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-27T09:32:37.728 INFO:teuthology.orchestra.run.smithi170.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-27T09:32:37.754 INFO:teuthology.orchestra.run.smithi170.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-27T09:32:37.779 INFO:teuthology.orchestra.run.smithi170.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-03-27T09:32:37.829 INFO:teuthology.orchestra.run.smithi170.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-27T09:32:37.888 INFO:teuthology.orchestra.run.smithi170.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-27T09:32:37.905 INFO:teuthology.orchestra.run.smithi170.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-27T09:32:39.472 INFO:teuthology.orchestra.run.smithi170.stdout:(36/36): ceph-osd-19.0.0-2247.gad2494b3.el9.x86 22 MB/s | 71 MB 00:03 2024-03-27T09:32:39.476 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:39.477 INFO:teuthology.orchestra.run.smithi170.stdout:Total 21 MB/s | 84 MB 00:04 2024-03-27T09:32:39.502 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:32:39.545 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:32:39.545 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:32:40.293 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:32:40.293 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:32:41.893 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:32:41.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T09:32:41.991 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T09:32:42.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T09:32:42.089 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T09:32:42.150 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T09:32:42.183 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T09:32:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T09:32:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T09:32:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T09:32:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T09:32:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T09:32:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T09:32:42.608 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T09:32:42.663 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T09:32:42.855 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T09:32:42.905 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T09:32:42.935 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T09:32:42.969 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T09:32:43.008 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T09:32:43.084 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T09:32:43.127 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T09:32:43.160 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T09:32:43.201 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T09:32:43.235 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T09:32:43.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T09:32:43.367 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T09:32:43.409 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T09:32:43.481 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T09:32:43.617 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T09:32:43.736 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T09:32:43.933 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9. 31/36 2024-03-27T09:32:43.957 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 32/36 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 32/36 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T09:32:43.982 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:45.780 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 33/36 2024-03-27T09:32:45.806 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 33/36 2024-03-27T09:32:45.807 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:45.807 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T09:32:45.807 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T09:32:45.807 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T09:32:45.807 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:46.347 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 34/36 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 34/36 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T09:32:46.371 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 35/36 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 35/36 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T09:32:46.637 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:46.711 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 36/36 2024-03-27T09:32:53.293 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 36/36 2024-03-27T09:32:53.293 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 1/36 2024-03-27T09:32:53.293 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 2/36 2024-03-27T09:32:53.293 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 3/36 2024-03-27T09:32:53.293 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 4/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 5/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9. 6/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T09:32:53.294 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T09:32:53.295 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T09:32:53.296 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T09:32:53.296 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T09:32:53.297 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T09:32:53.297 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T09:32:53.297 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T09:32:53.297 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T09:32:53.297 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T09:32:54.067 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T09:32:54.067 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:54.067 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:32:54.067 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:54.067 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T09:32:54.068 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T09:32:54.069 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:54.070 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:32:54.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2024-03-27T09:32:54.884 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:33 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:32:54.948 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 33/36 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 33/36 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T09:32:54.975 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:54.981 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:32:55.033 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:32:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:32:55.034 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:32:55.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2024-03-27T09:32:55.515 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 34/36 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 34/36 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T09:32:55.539 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:55.653 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:34 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:32:55.774 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 35/36 2024-03-27T09:32:55.797 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 35/36 2024-03-27T09:32:55.797 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:55.797 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T09:32:55.798 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T09:32:55.798 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T09:32:55.798 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:32:55.802 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:32:55.803 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 750 k 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 750 k 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 757 k 2024-03-27T09:32:55.804 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:32:55.864 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 36/36 2024-03-27T09:32:56.245 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-19.0.0-2247.gad2494b3.el9.noarch.rpm 1.7 MB/s | 750 kB 00:00 2024-03-27T09:32:56.245 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:56.246 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.7 MB/s | 750 kB 00:00 2024-03-27T09:32:56.246 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:32:56.251 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:32:56.251 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:32:56.257 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:32:56.258 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:32:56.335 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:32:56.728 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:32:56.825 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:32:57.204 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:32:57.510 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:32:57.511 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:57.511 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:32:57.511 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:32:57.511 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:57.511 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:32:57.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T09:32:58.295 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:37 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:32:58.446 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 157 k 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:32:58.447 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:32:58.448 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:32:58.448 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 157 k 2024-03-27T09:32:58.448 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 640 k 2024-03-27T09:32:58.448 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:32:58.758 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-19.0.0-2247.gad2494 506 kB/s | 157 kB 00:00 2024-03-27T09:32:58.759 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:32:58.759 INFO:teuthology.orchestra.run.smithi172.stdout:Total 504 kB/s | 157 kB 00:00 2024-03-27T09:32:58.759 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:32:58.769 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:32:58.769 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:32:58.821 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:32:58.822 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:32:58.958 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:32:59.017 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:32:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:32:59.042 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:32:59.042 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T09:32:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:00.013 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:00.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2024-03-27T09:33:00.825 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:39 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:00.923 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:00.975 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:00.976 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:00.977 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:01.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T09:33:01.595 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:40 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:01.744 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 1.8 M 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 34 k 2024-03-27T09:33:01.745 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 16 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2024-03-27T09:33:01.746 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:01.747 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 5.9 M 2024-03-27T09:33:01.747 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 23 M 2024-03-27T09:33:01.747 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:02.220 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2247.gad2 43 kB/s | 16 kB 00:00 2024-03-27T09:33:02.353 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2247.gad 66 kB/s | 34 kB 00:00 2024-03-27T09:33:02.378 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-27T09:33:02.790 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 36/36 2024-03-27T09:33:02.790 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 1/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 2/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 3/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 4/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 5/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9. 6/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T09:33:02.791 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T09:33:02.792 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T09:33:02.793 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T09:33:02.793 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T09:33:02.793 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T09:33:02.793 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T09:33:02.794 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T09:33:03.179 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): ceph-mgr-dashboard-19.0.0-2247.gad2494b 1.3 MB/s | 1.8 MB 00:01 2024-03-27T09:33:03.321 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:03.569 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T09:33:03.570 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T09:33:03.571 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T09:33:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T09:33:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T09:33:03.572 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:03.572 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:03.612 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 686 kB/s | 863 kB 00:01 2024-03-27T09:33:03.812 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2024-03-27T09:33:03.904 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 495 kB/s | 144 kB 00:00 2024-03-27T09:33:03.963 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 528 kB/s | 31 kB 00:00 2024-03-27T09:33:04.088 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-27T09:33:04.288 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-03-27T09:33:04.410 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:48 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:04.513 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:04.567 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:04.568 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:04.568 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:04.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2024-03-27T09:33:05.219 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:49 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:05.368 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:05.369 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:05.369 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:05.369 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:05.369 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 750 k 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 750 k 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 757 k 2024-03-27T09:33:05.370 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:05.794 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-19.0.0-2247.gad2494b3.el9.noarch.rpm 1.7 MB/s | 750 kB 00:00 2024-03-27T09:33:05.795 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:05.795 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.7 MB/s | 750 kB 00:00 2024-03-27T09:33:05.795 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:05.800 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:05.800 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:05.806 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:05.806 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:05.884 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:06.047 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 149 kB/s | 548 kB 00:03 2024-03-27T09:33:06.050 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:06.051 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.4 MB/s | 5.9 MB 00:04 2024-03-27T09:33:06.071 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:06.089 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:06.089 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:06.247 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:06.248 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:06.402 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:33:06.481 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:33:06.725 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:06.803 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-27T09:33:06.833 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-27T09:33:06.890 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:33:06.940 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-27T09:33:07.066 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-27T09:33:07.128 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/1 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:07.180 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:07.211 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-27T09:33:07.255 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-27T09:33:07.283 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-27T09:33:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9 9/11 2024-03-27T09:33:07.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T09:33:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el 10/11 2024-03-27T09:33:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 11/11 2024-03-27T09:33:07.984 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:51 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:08.138 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 157 k 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:08.139 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:08.140 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 157 k 2024-03-27T09:33:08.140 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 640 k 2024-03-27T09:33:08.140 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 11/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el 1/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 2/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9 3/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-27T09:33:08.381 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-27T09:33:08.442 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-19.0.0-2247.gad2494 519 kB/s | 157 kB 00:00 2024-03-27T09:33:08.443 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:08.443 INFO:teuthology.orchestra.run.smithi170.stdout:Total 517 kB/s | 157 kB 00:00 2024-03-27T09:33:08.443 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:08.453 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:08.453 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:08.506 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:08.506 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:08.650 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:08.710 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:33:08.734 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:08.735 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:33:08.736 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:33:08.736 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T09:33:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:08.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T09:33:09.532 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:48 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:09.684 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:09.685 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 7.4 M 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T09:33:09.686 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-03-27T09:33:09.687 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:09.688 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 37 M 2024-03-27T09:33:09.688 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 209 M 2024-03-27T09:33:09.688 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3. 1/1 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:09.700 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:09.918 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-27T09:33:09.952 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-27T09:33:09.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2024-03-27T09:33:09.995 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 832 kB/s | 34 kB 00:00 2024-03-27T09:33:10.086 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-27T09:33:10.286 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-03-27T09:33:10.353 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-27T09:33:10.378 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-27T09:33:10.537 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-22 9.8 MB/s | 7.4 MB 00:00 2024-03-27T09:33:10.567 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:54 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:10.667 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:10.719 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:10.720 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:10.720 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:10.745 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-27T09:33:10.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T09:33:11.271 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 467 kB/s | 245 kB 00:00 2024-03-27T09:33:11.355 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:55 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:11.504 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 202 kB/s | 195 kB 00:00 2024-03-27T09:33:11.506 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:11.507 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-27T09:33:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:11.507 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-27T09:33:11.507 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:11.507 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 1.8 M 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 34 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 16 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T09:33:11.508 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout:Install 11 Packages 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.9 M 2024-03-27T09:33:11.509 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 23 M 2024-03-27T09:33:11.510 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:11.888 INFO:teuthology.orchestra.run.smithi170.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2247.gad 159 kB/s | 34 kB 00:00 2024-03-27T09:33:11.913 INFO:teuthology.orchestra.run.smithi170.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2247.gad2 69 kB/s | 16 kB 00:00 2024-03-27T09:33:12.039 INFO:teuthology.orchestra.run.smithi170.stdout:(3/11): ceph-mgr-dashboard-19.0.0-2247.gad2494b 4.9 MB/s | 1.8 MB 00:00 2024-03-27T09:33:12.064 INFO:teuthology.orchestra.run.smithi170.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-27T09:33:12.196 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 443 kB/s | 803 kB 00:01 2024-03-27T09:33:12.199 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:12.200 INFO:teuthology.orchestra.run.smithi172.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-27T09:33:12.231 INFO:teuthology.orchestra.run.smithi170.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-27T09:33:12.244 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:12.270 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:12.270 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:12.281 INFO:teuthology.orchestra.run.smithi170.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-03-27T09:33:12.507 INFO:teuthology.orchestra.run.smithi170.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 642 kB/s | 144 kB 00:00 2024-03-27T09:33:12.532 INFO:teuthology.orchestra.run.smithi170.stdout:(8/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-27T09:33:12.627 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:12.628 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:12.632 INFO:teuthology.orchestra.run.smithi170.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.1 MB/s | 2.0 MB 00:00 2024-03-27T09:33:12.649 INFO:teuthology.orchestra.run.smithi170.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 216 kB/s | 31 kB 00:00 2024-03-27T09:33:12.674 INFO:teuthology.orchestra.run.smithi170.stdout:(11/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-27T09:33:12.679 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:12.679 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.0 MB/s | 5.9 MB 00:01 2024-03-27T09:33:12.697 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:12.714 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:12.877 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:12.877 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:13.349 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:13.394 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T09:33:13.425 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T09:33:13.430 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-27T09:33:13.461 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-27T09:33:13.519 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T09:33:13.554 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T09:33:13.570 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-27T09:33:13.702 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-27T09:33:13.730 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T09:33:13.748 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-27T09:33:13.760 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T09:33:13.823 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-27T09:33:13.846 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T09:33:13.875 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-27T09:33:13.903 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-27T09:33:13.935 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9 9/11 2024-03-27T09:33:14.181 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el 10/11 2024-03-27T09:33:14.210 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 11/11 2024-03-27T09:33:14.499 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T09:33:14.622 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T09:33:14.981 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 11/11 2024-03-27T09:33:14.981 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el 1/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noa 2/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9 3/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-27T09:33:14.982 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-27T09:33:15.329 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:15.330 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:15.331 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:15.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T09:33:15.850 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T09:33:16.166 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:00 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:16.319 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:16.320 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-27T09:33:16.320 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:16.320 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-27T09:33:16.320 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 7.4 M 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T09:33:16.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:16.322 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2024-03-27T09:33:16.323 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 209 M 2024-03-27T09:33:16.323 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:16.872 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 200 kB/s | 32 kB 00:00 2024-03-27T09:33:16.873 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T09:33:16.901 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 12/12 2024-03-27T09:33:16.906 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-03-27T09:33:16.948 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 838 kB/s | 34 kB 00:00 2024-03-27T09:33:16.998 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-27T09:33:17.199 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-27T09:33:17.257 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-27T09:33:17.282 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 217 kB 00:00 2024-03-27T09:33:17.400 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-22 11 MB/s | 7.4 MB 00:00 2024-03-27T09:33:17.675 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-27T09:33:17.809 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 481 kB/s | 195 kB 00:00 2024-03-27T09:33:17.833 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-27T09:33:17.909 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-27T09:33:17.911 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:17.911 INFO:teuthology.orchestra.run.smithi170.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-27T09:33:17.953 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:17.978 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:17.978 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:18.330 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:18.331 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:19.057 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:19.123 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T09:33:19.154 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T09:33:19.248 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T09:33:19.283 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T09:33:19.373 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 12/12 2024-03-27T09:33:19.373 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 1/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T09:33:19.374 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T09:33:19.453 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T09:33:19.482 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T09:33:19.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T09:33:19.810 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:19.811 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:20.203 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T09:33:20.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2024-03-27T09:33:20.311 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T09:33:20.792 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:59 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:20.941 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:20.942 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 49 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T09:33:20.943 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2024-03-27T09:33:20.944 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:20.945 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.0 M 2024-03-27T09:33:20.945 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-03-27T09:33:20.945 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:21.144 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2024-03-27T09:33:21.168 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-27T09:33:21.210 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-27T09:33:21.236 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-27T09:33:21.261 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-27T09:33:21.286 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-27T09:33:21.311 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): ceph-mgr-rook-19.0.0-2247.gad2494b3.el9 170 kB/s | 49 kB 00:00 2024-03-27T09:33:21.362 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-27T09:33:21.387 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-03-27T09:33:21.454 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-27T09:33:21.479 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-27T09:33:21.504 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-27T09:33:21.569 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T09:33:21.638 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-27T09:33:21.640 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:21.640 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-27T09:33:21.640 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:21.654 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:21.654 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:21.742 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:21.743 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:22.248 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:22.411 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T09:33:22.464 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T09:33:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T09:33:22.565 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T09:33:22.594 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T09:33:22.606 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T09:33:22.624 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 12/12 2024-03-27T09:33:22.709 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T09:33:22.817 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T09:33:22.874 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T09:33:23.592 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T09:33:23.647 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T09:33:23.680 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T09:33:23.728 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T09:33:23.750 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 13/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 13/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 1/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T09:33:24.501 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T09:33:24.502 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T09:33:24.848 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:24.849 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:25.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T09:33:25.391 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 12/12 2024-03-27T09:33:25.391 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2247.gad249 1/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T09:33:25.392 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T09:33:25.638 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:04 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:25.791 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:25.791 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 140 k 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================== 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-03-27T09:33:25.792 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:25.793 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.8 M 2024-03-27T09:33:25.793 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 33 M 2024-03-27T09:33:25.793 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:25.808 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T09:33:25.809 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T09:33:25.810 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T09:33:25.810 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:25.810 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:26.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2024-03-27T09:33:26.456 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-27T09:33:26.515 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 813 kB/s | 248 kB 00:00 2024-03-27T09:33:26.581 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2247.gad2494b3.e 376 kB/s | 140 kB 00:00 2024-03-27T09:33:26.723 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-03-27T09:33:26.790 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2024-03-27T09:33:26.812 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:10 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:26.824 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-27T09:33:26.825 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:26.825 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.6 MB/s | 6.8 MB 00:01 2024-03-27T09:33:26.839 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:26.850 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:26.850 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:26.938 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:26.938 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:26.964 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:26.965 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:26.965 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:26.965 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:26.965 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 49 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T09:33:26.966 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.0 M 2024-03-27T09:33:26.967 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-03-27T09:33:26.968 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:27.242 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T09:33:27.441 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2024-03-27T09:33:27.466 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-27T09:33:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T09:33:27.525 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-27T09:33:27.550 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-27T09:33:27.575 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-27T09:33:27.601 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-27T09:33:27.626 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): ceph-mgr-rook-19.0.0-2247.gad2494b3.el9 159 kB/s | 49 kB 00:00 2024-03-27T09:33:27.718 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-27T09:33:27.743 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-27T09:33:27.844 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 933 kB/s | 203 kB 00:00 2024-03-27T09:33:27.869 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 471 kB/s | 59 kB 00:00 2024-03-27T09:33:27.928 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-27T09:33:27.995 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-27T09:33:27.996 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:27.997 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-27T09:33:27.997 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:28.010 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:28.010 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T09:33:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T09:33:28.099 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:28.099 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:28.149 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T09:33:28.171 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 6/6 2024-03-27T09:33:28.614 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:28.784 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T09:33:28.829 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T09:33:28.884 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T09:33:28.947 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T09:33:28.988 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T09:33:28.990 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 6/6 2024-03-27T09:33:28.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/6 2024-03-27T09:33:28.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T09:33:28.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T09:33:28.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T09:33:28.991 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T09:33:29.083 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T09:33:29.191 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T09:33:29.240 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:29.291 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:29.292 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:29.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2024-03-27T09:33:29.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T09:33:30.012 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T09:33:30.046 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T09:33:30.085 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T09:33:30.095 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:09 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:30.107 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 13/13 2024-03-27T09:33:30.245 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 870 k 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:30.246 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 949 k 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 4.0 M 2024-03-27T09:33:30.247 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:30.377 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-03-27T09:33:30.811 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-19.0.0-2247.gad2494b3.el9.x86_ 1.5 MB/s | 870 kB 00:00 2024-03-27T09:33:30.813 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:30.813 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.6 MB/s | 949 kB 00:00 2024-03-27T09:33:30.813 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:30.834 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:30.835 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 13/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 1/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T09:33:30.902 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T09:33:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T09:33:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T09:33:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T09:33:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T09:33:30.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T09:33:30.907 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:30.907 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:31.095 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:31.237 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T09:33:31.253 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:31.254 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:31.303 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2/2 2024-03-27T09:33:31.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T09:33:32.017 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2/2 2024-03-27T09:33:32.017 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/2 2024-03-27T09:33:32.082 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:16 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:32.233 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 140 k 2024-03-27T09:33:32.234 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-03-27T09:33:32.235 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:32.236 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2024-03-27T09:33:32.236 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2024-03-27T09:33:32.236 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:32.348 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:32.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2024-03-27T09:33:32.747 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-27T09:33:32.889 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2247.gad2494b3.e 310 kB/s | 140 kB 00:00 2024-03-27T09:33:32.956 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 478 kB/s | 248 kB 00:00 2024-03-27T09:33:33.196 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:33.345 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:33.345 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:33.345 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-27T09:33:33.345 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:33.345 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 264 k 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-03-27T09:33:33.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:33.347 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 332 k 2024-03-27T09:33:33.347 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.5 M 2024-03-27T09:33:33.347 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:33.356 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 123 kB/s | 58 kB 00:00 2024-03-27T09:33:33.415 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 111 kB/s | 51 kB 00:00 2024-03-27T09:33:33.481 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 512 kB/s | 69 kB 00:00 2024-03-27T09:33:33.507 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 722 kB/s | 548 kB 00:00 2024-03-27T09:33:33.510 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:33.510 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-27T09:33:33.523 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:33.534 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:33.534 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:33.623 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:33.624 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:33.632 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-volume-19.0.0-2247.gad2494b3.el9.no 926 kB/s | 264 kB 00:00 2024-03-27T09:33:33.633 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:33.633 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-27T09:33:33.633 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:33.642 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:33.642 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:33.681 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:33.682 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:33.929 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:34.097 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T09:33:34.106 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T09:33:34.150 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T09:33:34.151 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2/2 2024-03-27T09:33:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2/2 2024-03-27T09:33:34.167 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:33:34.167 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T09:33:34.167 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T09:33:34.752 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T09:33:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T09:33:34.854 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 6/6 2024-03-27T09:33:34.897 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 1/2 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:35.247 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:35.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 6/6 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 1/6 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T09:33:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T09:33:36.019 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T09:33:36.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:36.019 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:36.020 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:36.147 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:15 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:36.294 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2024-03-27T09:33:36.296 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 125 k 2024-03-27T09:33:36.297 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 125 k 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 445 k 2024-03-27T09:33:36.298 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:36.512 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-19.0.0-2247.gad2494b3.el9.x86_64 584 kB/s | 125 kB 00:00 2024-03-27T09:33:36.513 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:36.513 INFO:teuthology.orchestra.run.smithi172.stdout:Total 580 kB/s | 125 kB 00:00 2024-03-27T09:33:36.513 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:36.519 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:36.519 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:36.576 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:36.576 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:36.720 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:36.827 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:36.866 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:20 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:37.018 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:37.018 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 870 k 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-03-27T09:33:37.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:37.020 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 949 k 2024-03-27T09:33:37.020 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 4.0 M 2024-03-27T09:33:37.020 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:37.151 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 594 kB/s | 79 kB 00:00 2024-03-27T09:33:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:37.485 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-19.0.0-2247.gad2494b3.el9.x86_ 1.8 MB/s | 870 kB 00:00 2024-03-27T09:33:37.486 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:37.486 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.0 MB/s | 949 kB 00:00 2024-03-27T09:33:37.486 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:37.508 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:37.508 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:37.561 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:37.583 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:37.583 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:37.779 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:37.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2024-03-27T09:33:37.929 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T09:33:38.009 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2/2 2024-03-27T09:33:38.403 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:17 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:38.504 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:38.560 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:38.561 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:38.562 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:38.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2024-03-27T09:33:38.774 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2/2 2024-03-27T09:33:38.774 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/2 2024-03-27T09:33:39.085 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T09:33:39.086 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:39.086 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:39.086 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T09:33:39.086 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:39.086 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:39.182 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:18 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:39.331 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 31 k 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:39.332 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 31 k 2024-03-27T09:33:39.333 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 140 k 2024-03-27T09:33:39.333 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:39.373 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2024-03-27T09:33:39.506 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-19.0.0-2247.gad2494b3.el9.x86_6 180 kB/s | 31 kB 00:00 2024-03-27T09:33:39.507 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:39.507 INFO:teuthology.orchestra.run.smithi172.stdout:Total 179 kB/s | 31 kB 00:00 2024-03-27T09:33:39.507 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:39.511 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:39.511 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:39.541 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:39.542 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:39.637 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:39.754 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:39.957 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:23 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:40.107 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:40.107 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-2247.gad2494b3.el9 ceph-noarch 264 k 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-03-27T09:33:40.108 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:40.109 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 332 k 2024-03-27T09:33:40.109 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.5 M 2024-03-27T09:33:40.109 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:40.121 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:40.232 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 561 kB/s | 69 kB 00:00 2024-03-27T09:33:40.365 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-volume-19.0.0-2247.gad2494b3.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-27T09:33:40.366 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:40.367 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-27T09:33:40.367 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:40.375 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:40.375 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:40.380 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:40.415 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:40.415 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:40.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2024-03-27T09:33:40.642 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:40.830 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T09:33:40.859 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2/2 2024-03-27T09:33:40.875 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2/2 2024-03-27T09:33:40.875 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:33:40.875 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T09:33:40.875 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:41.100 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:20 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:41.197 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:41.249 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:41.250 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:41.250 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:41.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2024-03-27T09:33:41.582 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 1/2 2024-03-27T09:33:41.875 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:20 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T09:33:41.903 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:41.903 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-2247.gad2494b3.el9.noarch 2024-03-27T09:33:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T09:33:41.904 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:41.904 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:41.972 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:42.025 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:42.026 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:42.026 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:42.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2024-03-27T09:33:42.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2024-03-27T09:33:42.655 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:21 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:42.756 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:42.790 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:26 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:42.809 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:42.810 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:42.811 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:42.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2024-03-27T09:33:42.940 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 125 k 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:42.941 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:42.942 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:42.942 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:42.942 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 125 k 2024-03-27T09:33:42.942 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 445 k 2024-03-27T09:33:42.942 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:43.157 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-19.0.0-2247.gad2494b3.el9.x86_64 581 kB/s | 125 kB 00:00 2024-03-27T09:33:43.157 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:43.158 INFO:teuthology.orchestra.run.smithi170.stdout:Total 577 kB/s | 125 kB 00:00 2024-03-27T09:33:43.158 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:43.163 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:43.163 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:43.218 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:43.218 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:43.361 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:43.442 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:22 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:43.464 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:43.538 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:43.589 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:43.590 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:43.590 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:43.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2024-03-27T09:33:43.896 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:44.199 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:44.200 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:44.200 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:44.200 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:44.210 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:23 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:44.307 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:44.359 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:44.360 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:44.360 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:44.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2024-03-27T09:33:44.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2024-03-27T09:33:44.991 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:23 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:45.025 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:29 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:45.088 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:45.125 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:45.140 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:45.141 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-03-27T09:33:45.141 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:45.179 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:45.180 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:45.180 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:45.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2024-03-27T09:33:45.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2024-03-27T09:33:45.774 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:24 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:45.831 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:29 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:45.923 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 90 k 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:33:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:45.925 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 90 k 2024-03-27T09:33:45.925 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 313 k 2024-03-27T09:33:45.925 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:45.980 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 31 k 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:45.981 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:45.982 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:45.982 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 k 2024-03-27T09:33:45.982 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 140 k 2024-03-27T09:33:45.982 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:46.134 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-19.0.0-2247.gad2494b3.el9.x86_64.rpm 432 kB/s | 90 kB 00:00 2024-03-27T09:33:46.134 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:46.135 INFO:teuthology.orchestra.run.smithi172.stdout:Total 429 kB/s | 90 kB 00:00 2024-03-27T09:33:46.135 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:46.143 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:46.143 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:46.144 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-19.0.0-2247.gad2494b3.el9.x86_6 193 kB/s | 31 kB 00:00 2024-03-27T09:33:46.144 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:46.145 INFO:teuthology.orchestra.run.smithi170.stdout:Total 192 kB/s | 31 kB 00:00 2024-03-27T09:33:46.145 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:46.148 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:46.148 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:46.180 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:46.180 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:46.198 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:46.198 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:46.263 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:46.339 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:46.385 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:46.440 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:46.807 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:46.813 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:47.082 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:47.082 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:47.083 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:47.083 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:47.083 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:47.083 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:47.098 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:47.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2024-03-27T09:33:47.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2024-03-27T09:33:47.839 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:31 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:47.906 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:26 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:47.936 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:47.990 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:47.991 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:47.991 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:48.066 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.4 M 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:48.067 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.4 M 2024-03-27T09:33:48.068 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 29 M 2024-03-27T09:33:48.068 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:48.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2024-03-27T09:33:48.445 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-19.0.0-2247.gad2494b3.el9.x86_64.rpm 9.0 MB/s | 3.4 MB 00:00 2024-03-27T09:33:48.445 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:48.446 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.0 MB/s | 3.4 MB 00:00 2024-03-27T09:33:48.446 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:48.457 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:48.457 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:48.536 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:48.536 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:48.685 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:32 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:48.782 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:48.834 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:48.835 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:48.835 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:48.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2024-03-27T09:33:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:49.196 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:33:49.196 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T09:33:49.196 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T09:33:49.196 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T09:33:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:49.482 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:33 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:49.578 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:49.629 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:49.630 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:49.703 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2024-03-27T09:33:50.268 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:34 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:50.364 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:50.415 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:50.416 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:50.416 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:50.430 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:50.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2024-03-27T09:33:50.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2024-03-27T09:33:51.055 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:35 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:51.151 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:51.202 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:51.203 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:51.203 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:51.267 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:30 ago on Wed 27 Mar 2024 09:31:21 AM UTC. 2024-03-27T09:33:51.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2024-03-27T09:33:51.416 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 175 k 2024-03-27T09:33:51.417 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 175 k 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 673 k 2024-03-27T09:33:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-27T09:33:51.773 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-19.0.0-2247.gad2494b3.el9.x86_64.rpm 495 kB/s | 175 kB 00:00 2024-03-27T09:33:51.774 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:51.774 INFO:teuthology.orchestra.run.smithi172.stdout:Total 492 kB/s | 175 kB 00:00 2024-03-27T09:33:51.774 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-27T09:33:51.784 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-27T09:33:51.784 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-27T09:33:51.838 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:35 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:51.840 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-27T09:33:51.840 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-27T09:33:51.939 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:19.0.0-2247.gad2494b3.el9.x86_64 is already installed. 2024-03-27T09:33:51.990 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-27T09:33:51.990 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:51.992 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-03-27T09:33:51.992 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:52.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2024-03-27T09:33:52.088 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:52.470 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:52.628 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:36 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:52.777 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:52.777 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 90 k 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 90 k 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 313 k 2024-03-27T09:33:52.778 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:52.780 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:52.781 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-27T09:33:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:52.781 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:33:52.781 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-27T09:33:52.989 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-19.0.0-2247.gad2494b3.el9.x86_64.rpm 429 kB/s | 90 kB 00:00 2024-03-27T09:33:52.989 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:52.990 INFO:teuthology.orchestra.run.smithi170.stdout:Total 426 kB/s | 90 kB 00:00 2024-03-27T09:33:52.990 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:52.998 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:52.998 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:53.028 DEBUG:teuthology.parallel:result is None 2024-03-27T09:33:53.054 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:53.054 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:53.188 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:53.287 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:53.674 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:53.958 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:54.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2024-03-27T09:33:54.784 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:38 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:54.938 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 3.4 M 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.4 M 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 29 M 2024-03-27T09:33:54.939 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:55.984 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-19.0.0-2247.gad2494b3.el9.x86_64.rpm 3.3 MB/s | 3.4 MB 00:01 2024-03-27T09:33:55.985 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:55.985 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.3 MB/s | 3.4 MB 00:01 2024-03-27T09:33:55.985 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:55.996 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:55.996 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:56.072 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:56.072 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:56.644 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:56.710 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T09:33:56.734 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:57.923 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:33:58.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2024-03-27T09:33:58.770 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:42 ago on Wed 27 Mar 2024 09:31:16 AM UTC. 2024-03-27T09:33:58.920 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-2247.gad2494b3.el9 ceph 175 k 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 175 k 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 673 k 2024-03-27T09:33:58.921 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-27T09:33:59.178 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-19.0.0-2247.gad2494b3.el9.x86_64.rpm 683 kB/s | 175 kB 00:00 2024-03-27T09:33:59.179 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-27T09:33:59.179 INFO:teuthology.orchestra.run.smithi170.stdout:Total 679 kB/s | 175 kB 00:00 2024-03-27T09:33:59.179 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-27T09:33:59.189 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-27T09:33:59.189 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-27T09:33:59.246 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-27T09:33:59.246 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-27T09:33:59.390 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-27T09:33:59.495 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:33:59.950 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 1/1 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-2247.gad2494b3.el9.x86_64 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:00.235 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-27T09:34:00.529 DEBUG:teuthology.parallel:result is None 2024-03-27T09:34:00.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ad2494b32de08d5bf12408a9a641b9ebfb123d09 2024-03-27T09:34:00.672 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T09:34:00.693 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-2247.gad2494b3.el9 2024-03-27T09:34:00.694 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2247.gad2494b3.el9 2024-03-27T09:34:00.694 INFO:teuthology.task.install:The correct ceph version 19.0.0-2247.gad2494b3 is installed. 2024-03-27T09:34:00.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ad2494b32de08d5bf12408a9a641b9ebfb123d09 2024-03-27T09:34:00.911 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T09:34:00.936 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-2247.gad2494b3.el9 2024-03-27T09:34:00.936 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2247.gad2494b3.el9 2024-03-27T09:34:00.936 INFO:teuthology.task.install:The correct ceph version 19.0.0-2247.gad2494b3 is installed. 2024-03-27T09:34:00.938 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T09:34:00.938 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:00.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T09:34:00.967 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:00.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T09:34:01.007 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T09:34:01.007 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:01.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T09:34:01.036 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T09:34:01.103 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:01.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T09:34:01.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T09:34:01.204 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T09:34:01.204 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:01.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T09:34:01.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T09:34:01.301 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:01.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T09:34:01.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T09:34:01.399 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T09:34:01.399 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:01.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T09:34:01.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T09:34:01.493 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:01.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T09:34:01.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T09:34:01.590 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T09:34:01.680 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T09:34:01.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2024-03-27T09:34:01.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-03-27T09:34:01.710 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T09:34:01.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T09:34:01.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T09:34:01.778 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T09:34:01.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T09:34:01.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T09:34:01.854 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T09:34:01.855 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-27T09:34:01.855 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/7625965', 'branch': 'wip-sjust-crimson-testing-2024-03-20', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized}', 'email': None, 'first_in_suite': False, 'job_id': '7625965', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-sjust-crimson-testing-2024-03-20'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-thrash-osd', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6db07181a4df6107bd2ae48adf3d2421c02fa2c2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8843, 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-thrash-osd', 'suite_path': '/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/Matan-B/ceph.git', 'suite_sha1': '6db07181a4df6107bd2ae48adf3d2421c02fa2c2', 'targets': {'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLxkoIjMjUjtIe7Wmc5+7Wr2fHSgfxRSnOQ7Ih8nYqPiMa6fdfWXC9Hao0kOmzN3OmyR7icp1d20917tGueayzg=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbyDdzKHVWfMa4Qf1LA5apN8ds2mpDxV5Qmfe4sXxBRhk8MyiGkZjKsleV3XsqJ6tgYL4+MCbMbFDDS29cPE4c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'localize_reads': True, 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_08:49:03', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T09:34:01.855 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T09:34:01.885 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T09:34:01.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T09:34:01.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T09:34:01.967 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:01.967 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T09:34:02.019 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T09:34:02.020 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-27 09:33:56.985263093 +0000 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-27 09:30:28.391376477 +0000 2024-03-27T09:34:02.074 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-27 09:30:28.391376477 +0000 2024-03-27T09:34:02.075 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-27 09:30:28.391376477 +0000 2024-03-27T09:34:02.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T09:34:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-27T09:34:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-27T09:34:02.140 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000101771 s, 5.0 MB/s 2024-03-27T09:34:02.141 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T09:34:02.197 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-27 09:33:56.985263093 +0000 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-27 09:30:28.721368581 +0000 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-27 09:30:28.721368581 +0000 2024-03-27T09:34:02.252 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-27 09:30:28.721368581 +0000 2024-03-27T09:34:02.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T09:34:02.317 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-27T09:34:02.317 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-27T09:34:02.318 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.0572e-05 s, 6.4 MB/s 2024-03-27T09:34:02.319 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T09:34:02.374 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-27T09:34:02.428 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T09:34:02.428 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.428 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-27T09:34:02.428 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.428 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.429 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-27 09:33:56.986263070 +0000 2024-03-27T09:34:02.429 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-27 09:30:29.054360613 +0000 2024-03-27T09:34:02.429 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-27 09:30:29.054360613 +0000 2024-03-27T09:34:02.429 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-27 09:30:29.054360613 +0000 2024-03-27T09:34:02.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T09:34:02.493 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-27T09:34:02.494 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-27T09:34:02.494 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 7.3993e-05 s, 6.9 MB/s 2024-03-27T09:34:02.495 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T09:34:02.549 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-27T09:34:02.604 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T09:34:02.604 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.604 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-27T09:34:02.604 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.604 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.605 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-27 09:33:56.986263070 +0000 2024-03-27T09:34:02.605 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-27 09:30:29.571348242 +0000 2024-03-27T09:34:02.605 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-27 09:30:29.571348242 +0000 2024-03-27T09:34:02.605 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-27 09:30:29.571348242 +0000 2024-03-27T09:34:02.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T09:34:02.669 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-27T09:34:02.669 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-27T09:34:02.670 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 7.967e-05 s, 6.4 MB/s 2024-03-27T09:34:02.670 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T09:34:02.726 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:02.726 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:02.726 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T09:34:02.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T09:34:02.744 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-27 09:33:49.436196217 +0000 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-27 09:30:28.648808861 +0000 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-27 09:30:28.648808861 +0000 2024-03-27T09:34:02.802 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-27 09:30:28.648808861 +0000 2024-03-27T09:34:02.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T09:34:02.870 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-27T09:34:02.871 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-27T09:34:02.871 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000274772 s, 1.9 MB/s 2024-03-27T09:34:02.872 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T09:34:02.930 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-27 09:33:49.436196217 +0000 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-27 09:30:29.001800436 +0000 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-27 09:30:29.001800436 +0000 2024-03-27T09:34:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-27 09:30:29.001800436 +0000 2024-03-27T09:34:02.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T09:34:03.054 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-27T09:34:03.055 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-27T09:34:03.055 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000229464 s, 2.2 MB/s 2024-03-27T09:34:03.056 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T09:34:03.113 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:03.169 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-27 09:33:49.437196193 +0000 2024-03-27T09:34:03.170 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-27 09:30:29.340792346 +0000 2024-03-27T09:34:03.170 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-27 09:30:29.340792346 +0000 2024-03-27T09:34:03.170 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-27 09:30:29.340792346 +0000 2024-03-27T09:34:03.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T09:34:03.236 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-27T09:34:03.236 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-27T09:34:03.236 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000208944 s, 2.5 MB/s 2024-03-27T09:34:03.237 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T09:34:03.294 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-27T09:34:03.349 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-27 09:33:49.437196193 +0000 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-27 09:30:29.672784423 +0000 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-27 09:30:29.672784423 +0000 2024-03-27T09:34:03.350 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-27 09:30:29.672784423 +0000 2024-03-27T09:34:03.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T09:34:03.417 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-27T09:34:03.417 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-27T09:34:03.417 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00019518 s, 2.6 MB/s 2024-03-27T09:34:03.418 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T09:34:03.475 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:03.475 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-03-27T09:34:03.475 INFO:tasks.ceph:Generating config... 2024-03-27T09:34:03.476 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-27T09:34:03.476 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-27T09:34:03.476 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-27T09:34:03.476 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-27T09:34:03.476 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] log to file = False 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-27T09:34:03.477 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-03-27T09:34:03.478 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-27T09:34:03.479 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-03-27T09:34:03.479 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-03-27T09:34:03.479 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T09:34:03.479 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T09:34:03.479 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-27T09:34:03.479 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-27T09:34:03.480 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-27T09:34:03.480 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.480 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.480 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.480 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.480 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T09:34:03.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T09:34:03.548 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T09:34:03.551 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T09:34:03.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T09:34:03.633 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.170')] 2024-03-27T09:34:03.633 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.170', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-27T09:34:03.634 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:03.634 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T09:34:03.687 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.170 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:03.777 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:03.776+0000 7f8b80828f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:03.777 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:03.776+0000 7f8b80828f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:03.777 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:03.776+0000 7f8b80828f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: generated fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:setting min_mon_release = quincy 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 0 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:last_changed 2024-03-27T09:34:03.777835+0000 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:created 2024-03-27T09:34:03.777835+0000 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:min_mon_release 17 (quincy) 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:election_strategy: 1 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:0: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.a 2024-03-27T09:34:03.778 INFO:teuthology.orchestra.run.smithi170.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-27T09:34:03.779 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T09:34:03.794 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2512214e-2ed3-4e43-8731-25b1a524cd43... 2024-03-27T09:34:03.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T09:34:03.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T09:34:03.884 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2024-03-27T09:34:03.884 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T09:34:03.885 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2024-03-27T09:34:03.886 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = 172.21.15.170 2024-03-27T09:34:03.887 INFO:teuthology.orchestra.run.smithi170.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms client mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms cluster mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms mon client mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms mon cluster mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms mon service mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: ms service mode = crc 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_min_size = 2 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_size = 3 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 120 2024-03-27T09:34:03.888 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.889 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression algorithm = snappy 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression mode = aggressive 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore rocksdb cf = False 2024-03-27T09:34:03.890 INFO:teuthology.orchestra.run.smithi170.stdout: crimson alien thread cpu cores = 6-7 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: crimson osd obc lru size = 10 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 20 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 20 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: err to stderr = True 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: log flush on exit = True 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: log to file = False 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: log to stderr = True 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: osd blocked scrub grace period = 3600 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug reject backfill probability = 0.3 2024-03-27T09:34:03.891 INFO:teuthology.orchestra.run.smithi170.stdout: osd delete sleep = 1 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd max backfills = 3 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd max markdown count = 1000 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub min interval = 60 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd shutdown pgref assert = True 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: osd snap trim sleep = 2 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-27T09:34:03.892 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2024-03-27T09:34:03.893 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2024-03-27T09:34:03.894 INFO:teuthology.orchestra.run.smithi170.stdout: mon min osdmap epochs = 50 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune interval = 2 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune min = 15 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: mon osdmap full prune txsize = 2 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default crimson = True 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: paxos service trim min = 10 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2024-03-27T09:34:03.895 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout: crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout: crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout: crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi170.stdout: crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.896 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-03-27T09:34:03.897 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-03-27T09:34:03.898 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-03-27T09:34:03.899 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.170 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms client mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms cluster mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon client mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon cluster mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon service mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: ms service mode = crc 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_min_size = 2 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_size = 3 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:03.900 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 120 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-03-27T09:34:03.901 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.902 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = snappy 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore rocksdb cf = False 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: crimson alien thread cpu cores = 6-7 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: crimson osd obc lru size = 10 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-03-27T09:34:03.903 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: err to stderr = True 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: log flush on exit = True 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: log to file = False 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: log to stderr = True 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug reject backfill probability = 0.3 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd delete sleep = 1 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd max backfills = 3 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-03-27T09:34:03.904 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub min interval = 60 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: osd snap trim sleep = 2 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-03-27T09:34:03.905 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.906 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2024-03-27T09:34:03.907 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default crimson = True 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-03-27T09:34:03.908 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-03-27T09:34:03.909 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-03-27T09:34:03.910 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T09:34:03.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T09:34:03.975 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T09:34:03.975 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:03.975 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T09:34:03.990 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:03.990 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T09:34:04.044 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2024-03-27T09:34:04.044 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:04.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T09:34:04.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T09:34:04.123 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:04.123 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:04.178 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-03-27T09:34:04.178 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:04.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T09:34:04.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T09:34:04.220 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:04.220 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:04.276 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T09:34:04.276 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T09:34:04.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T09:34:04.393 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T09:34:04.396 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T09:34:04.397 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T09:34:04.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T09:34:04.448 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T09:34:04.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-27T09:34:04.515 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-27T09:34:04.531 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T09:34:04.531 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-03-27T09:34:04.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T09:34:04.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:04.557 INFO:tasks.ceph:role: osd.0 2024-03-27T09:34:04.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-27T09:34:04.558 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T09:34:04.623 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-27T09:34:04.624 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T09:34:04.624 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T09:34:04.624 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T09:34:04.624 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T09:34:04.656 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-27T09:34:04.657 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-27T09:34:04.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T09:34:04.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T09:34:04.800 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T09:34:04.867 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:04.868 INFO:tasks.ceph:role: osd.1 2024-03-27T09:34:04.868 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2024-03-27T09:34:04.868 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2024-03-27T09:34:04.934 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T09:34:04.935 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T09:34:04.935 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T09:34:04.935 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T09:34:04.965 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2024-03-27T09:34:04.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2024-03-27T09:34:04.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T09:34:05.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T09:34:05.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:05.451 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:05.452 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:05,453 seastar - Reactor backend: linux-aio 2024-03-27T09:34:05.468 INFO:teuthology.orchestra.run.smithi170.stderr:==28838==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:05.614 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:05,614 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T09:34:05.614 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:05,614 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-03-27T09:34:05.615 INFO:teuthology.orchestra.run.smithi170.stderr:DEBUG 2024-03-27 09:34:05,615 [shard 0:main] seastar - reactor::drain 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr:Suppressions used: 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr: count bytes template 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:06.187 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-27T09:34:06.258 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:06.262 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:06,262 seastar - Reactor backend: linux-aio 2024-03-27T09:34:06.274 INFO:teuthology.orchestra.run.smithi170.stderr:==28833==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr:Suppressions used: 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr: count bytes template 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:10.937 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-27T09:34:10.938 INFO:teuthology.orchestra.run.smithi170.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:10.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T09:34:10.988 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:11.272 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:11.273 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:11,273 seastar - Reactor backend: linux-aio 2024-03-27T09:34:11.278 INFO:teuthology.orchestra.run.smithi170.stderr:==29028==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:11.402 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:11,402 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T09:34:11.402 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:11,402 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-03-27T09:34:11.403 INFO:teuthology.orchestra.run.smithi170.stderr:DEBUG 2024-03-27 09:34:11,403 [shard 0:main] seastar - reactor::drain 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr:Suppressions used: 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr: count bytes template 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:11.932 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-27T09:34:11.988 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:11.992 INFO:teuthology.orchestra.run.smithi170.stderr:INFO 2024-03-27 09:34:11,992 seastar - Reactor backend: linux-aio 2024-03-27T09:34:12.006 INFO:teuthology.orchestra.run.smithi170.stderr:==29026==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:16.669 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:16.669 INFO:teuthology.orchestra.run.smithi170.stderr:Suppressions used: 2024-03-27T09:34:16.669 INFO:teuthology.orchestra.run.smithi170.stderr: count bytes template 2024-03-27T09:34:16.669 INFO:teuthology.orchestra.run.smithi170.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:16.670 INFO:teuthology.orchestra.run.smithi170.stderr:----------------------------------------------------- 2024-03-27T09:34:16.670 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-27T09:34:16.670 INFO:teuthology.orchestra.run.smithi170.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:16.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T09:34:16.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T09:34:16.751 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:16.751 INFO:tasks.ceph:role: osd.2 2024-03-27T09:34:16.751 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-27T09:34:16.751 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T09:34:16.830 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T09:34:16.831 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T09:34:16.831 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T09:34:17.341 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-27T09:34:17.343 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-27T09:34:17.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-03-27T09:34:17.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T09:34:17.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-27T09:34:17.585 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-27T09:34:17.585 INFO:tasks.ceph:role: osd.3 2024-03-27T09:34:17.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-27T09:34:17.585 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T09:34:17.659 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T09:34:17.660 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T09:34:18.105 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-27T09:34:18.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-27T09:34:18.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-03-27T09:34:18.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-27T09:34:18.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:18.548 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:18.549 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:18,550 seastar - Reactor backend: linux-aio 2024-03-27T09:34:18.562 INFO:teuthology.orchestra.run.smithi172.stderr:==29400==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:18,686 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T09:34:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:18,686 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-03-27T09:34:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-03-27 09:34:18,687 [shard 0:main] seastar - reactor::drain 2024-03-27T09:34:19.215 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:19.216 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-03-27T09:34:19.216 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-03-27T09:34:19.216 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:19.216 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:19.216 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-27T09:34:19.278 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:19.282 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:19,282 seastar - Reactor backend: linux-aio 2024-03-27T09:34:19.292 INFO:teuthology.orchestra.run.smithi172.stderr:==29398==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:19.357 INFO:teuthology.orchestra.run.smithi172.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb98b579 0xb85e040 0xb85f5bc 0xb85f7d2 0xb85f928 0xb85fdf1 0x54daf 0x5d6ddca 0x5d6dfba 0x5d6e546 0x5bb330d 0x5bce3d1 0x6ea80e4 0x6eb98b7 0x6ebad54 0x6ebadd9 0x6ebaedb 0x6ebafd3 0x6ebb1e0 0x6ebb3c6 0x6ebb715 0x6ebbab9 0x6ebbce9 0x6ebbf53 0x6ebc220 0x6ebc43e 0x6ebc4f6 0x6ebc5bc 0x6ebc69d 0x5a19924 0x5a19a2e 0x5a19aea 0x5a19ba6 0x5a19c62 0x5a19daf 0x5a19f3a 0x5a1a2fb 0x5a36d78 0xb870cc5 0xb88af5f 0xb92bf5d 0xb92d5ab 0xb609d13 0xb60a68f 0x367d57a 0x3feaf 0x3ff5f 0x3469434 2024-03-27T09:34:19.357 INFO:teuthology.orchestra.run.smithi172.stderr:kernel callstack: 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:23.948 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-27T09:34:23.949 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:23.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T09:34:23.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:24.288 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:24.289 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:24,290 seastar - Reactor backend: linux-aio 2024-03-27T09:34:24.294 INFO:teuthology.orchestra.run.smithi172.stderr:==29583==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:24,418 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-27T09:34:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:24,418 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-03-27T09:34:24.418 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-03-27 09:34:24,419 [shard 0:main] seastar - reactor::drain 2024-03-27T09:34:24.942 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:24.943 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-03-27T09:34:24.943 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-03-27T09:34:24.943 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:24.943 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:24.943 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-27T09:34:25.006 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T09:34:25.010 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-03-27 09:34:25,010 seastar - Reactor backend: linux-aio 2024-03-27T09:34:25.019 INFO:teuthology.orchestra.run.smithi172.stderr:==29581==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-27T09:34:29.665 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:29.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-27T09:34:29.714 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T09:34:29.714 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:29.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T09:34:29.745 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:29.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T09:34:29.816 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:29.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T09:34:29.843 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:29.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T09:34:29.910 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:29.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-27T09:34:29.977 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:29.977 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T09:34:29.995 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:29.995 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-27T09:34:30.032 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T09:34:30.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T09:34:30.063 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.0] 2024-03-27T09:34:30.063 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAO6ANmjrf9GRAA2YZHlDXIePhiKZ4GvKb+dg== 2024-03-27T09:34:30.063 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.1] 2024-03-27T09:34:30.063 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAU6ANm1oMVCRAAaS3yamRIYHjuTuVi5XNB3g== 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr.x] 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAM6ANmM7t7FxAANg9Z07w3jgKdtWA4rZRD2A== 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.2] 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAb6ANmjxM0GhAA0VXbN55Qxwrvpj2/MGOIeA== 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout:[osd.3] 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAh6ANmfy4RChAAJ4+11xzmViYWAgAg1nFvbg== 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout:[client.0] 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAM6ANm3X27GhAA+qfOR8l+Ij9rjjx1EAznaA== 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-03-27T09:34:30.064 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQAM6ANm6eS5HhAA80KlmBM4wBzblmr5mPpwMw== 2024-03-27T09:34:30.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T09:34:30.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T09:34:30.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T09:34:30.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T09:34:30.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T09:34:30.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T09:34:30.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T09:34:30.551 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T09:34:30.551 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T09:34:30.577 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T09:34:30.690 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:30.690+0000 7fa8d88b4a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.690 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:30.690+0000 7fa8d88b4a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T09:34:30.730 DEBUG:teuthology.orchestra.run.smithi170:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T09:34:30.785 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T09:34:30.785 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T09:34:30.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T09:34:30.827 INFO:tasks.ceph.mon.a:Started 2024-03-27T09:34:30.827 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T09:34:30.828 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T09:34:30.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T09:34:30.830 INFO:tasks.ceph.mgr.x:Started 2024-03-27T09:34:30.830 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T09:34:30.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T09:34:30.910 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:34:30.909+0000 7f66e94a0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.910 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:34:30.909+0000 7f66e94a0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.910 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:34:30.910+0000 7f66e94a0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.941 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:30.935+0000 7f681795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:30.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:30.967+0000 7f681795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.038 INFO:teuthology.orchestra.run.smithi170.stderr:adjusted tunables profile to default 2024-03-27T09:34:31.049 INFO:tasks.ceph:check_enable_crimson: True 2024-03-27T09:34:31.049 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-27T09:34:31.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-27T09:34:31.157 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:31.151+0000 7fa78379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.161 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:31.160+0000 7fa78379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.229 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:31.228+0000 7f15c976e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.233 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:31.233+0000 7f15c976e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.233 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:31.233+0000 7f15c976e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:31.239 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:34:31.239+0000 7f66e01f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-27T09:34:31.239 INFO:tasks.ceph.mon.a.smithi170.stderr:Please be aware that this feature is experimental, untested, 2024-03-27T09:34:31.240 INFO:tasks.ceph.mon.a.smithi170.stderr:unsupported, and may result in data corruption, data loss, 2024-03-27T09:34:31.240 INFO:tasks.ceph.mon.a.smithi170.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-27T09:34:31.240 INFO:tasks.ceph.mon.a.smithi170.stderr:feature with important data. 2024-03-27T09:34:31.240 INFO:tasks.ceph.mon.a.smithi170.stderr: 2024-03-27T09:34:31.710 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:31.709+0000 7f15c976e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T09:34:31.763 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:31.763+0000 7f15c976e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T09:34:32.051 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T09:34:32.051 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:32.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T09:34:32.079 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-27T09:34:32.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T09:34:32.145 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:32.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T09:34:32.173 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-27T09:34:32.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-27T09:34:32.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 22407295-246e-48a3-976e-795ef8d66974 0 2024-03-27T09:34:32.304 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.303+0000 7f15c976e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T09:34:32.362 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.362+0000 7f15c976e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T09:34:32.392 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.386+0000 7f0438314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.395 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.395+0000 7f0438314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.409 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.409+0000 7f15c976e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T09:34:32.465 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.465+0000 7f15c976e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T09:34:32.475 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-03-27T09:34:32.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new bfeb4b6f-ce32-4f7f-b280-31c298b1ced3 1 2024-03-27T09:34:32.511 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.510+0000 7f15c976e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T09:34:32.620 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.614+0000 7fd318a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.623 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.623+0000 7fd318a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.635 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.634+0000 7f15c976e000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T09:34:32.702 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-03-27T09:34:32.708 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.707+0000 7f15c976e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T09:34:32.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1 2 2024-03-27T09:34:32.758 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.757+0000 7f15c976e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T09:34:32.815 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.814+0000 7f15c976e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T09:34:32.825 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.818+0000 7f583995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.827 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:32.827+0000 7f583995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:32.864 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:32.864+0000 7f15c976e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T09:34:32.907 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-03-27T09:34:32.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 77bd27c5-42c4-4560-973c-39a14c1d746b 3 2024-03-27T09:34:33.028 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:33.021+0000 7fba66ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:33.031 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-27T09:34:33.030+0000 7fba66ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:33.110 INFO:teuthology.orchestra.run.smithi172.stdout:3 2024-03-27T09:34:33.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-27T09:34:33.158 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T09:34:33.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-27T09:34:33.201 INFO:tasks.ceph.osd.0:Started 2024-03-27T09:34:33.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-27T09:34:33.236 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T09:34:33.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-27T09:34:33.258 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.257+0000 7f15c976e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T09:34:33.278 INFO:tasks.ceph.osd.1:Started 2024-03-27T09:34:33.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-27T09:34:33.310 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T09:34:33.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-27T09:34:33.353 INFO:tasks.ceph.osd.2:Started 2024-03-27T09:34:33.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-03-27T09:34:33.384 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-27T09:34:33.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-03-27T09:34:33.396 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.395+0000 7f15c976e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T09:34:33.397 INFO:tasks.ceph.osd.3:Started 2024-03-27T09:34:33.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T09:34:33.518 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:33.512+0000 7fd9a371d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:33.522 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:33.522+0000 7fd9a371d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:33.530 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.529+0000 7f15c976e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T09:34:33.583 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.582+0000 7f15c976e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T09:34:33.591 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:33.591 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":7,"fsid":"2512214e-2ed3-4e43-8731-25b1a524cd43","created":"2024-03-27T09:34:30.956471+0000","modified":"2024-03-27T09:34:33.109731+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T09:34:33.109731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"22407295-246e-48a3-976e-795ef8d66974","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfeb4b6f-ce32-4f7f-b280-31c298b1ced3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"77bd27c5-42c4-4560-973c-39a14c1d746b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T09:34:33.603 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T09:34:33.603 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T09:34:33.705 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.704+0000 7f15c976e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T09:34:33.807 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.806+0000 7f15c976e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T09:34:33.973 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:33.973+0000 7f15c976e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T09:34:34.006 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T09:34:34.021 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:34.020+0000 7f15c976e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T09:34:34.110 INFO:teuthology.misc.health.smithi170.stderr:2024-03-27T09:34:34.105+0000 7f023dfe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:34.114 INFO:teuthology.misc.health.smithi170.stderr:2024-03-27T09:34:34.113+0000 7f023dfe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:34.183 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-27T09:34:34.183 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":7,"fsid":"2512214e-2ed3-4e43-8731-25b1a524cd43","created":"2024-03-27T09:34:30.956471+0000","modified":"2024-03-27T09:34:33.109731+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T09:34:33.109731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"22407295-246e-48a3-976e-795ef8d66974","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfeb4b6f-ce32-4f7f-b280-31c298b1ced3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"77bd27c5-42c4-4560-973c-39a14c1d746b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T09:34:34.191 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:34.191+0000 7f15c976e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T09:34:34.193 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-03-27T09:34:35.042 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:35.041+0000 7f15c976e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T09:34:35.512 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:35.511+0000 7f15c976e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T09:34:35.565 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:35.565+0000 7f15c976e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T09:34:35.692 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:35.692+0000 7f15c976e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T09:34:36.713 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:34:36.712+0000 7f15688dc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-27T09:34:40.597 DEBUG:teuthology.orchestra.run.smithi170:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T09:34:40.706 INFO:teuthology.misc.health.smithi170.stderr:2024-03-27T09:34:40.702+0000 7fc5d466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:40.710 INFO:teuthology.misc.health.smithi170.stderr:2024-03-27T09:34:40.710+0000 7fc5d466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:40.898 INFO:teuthology.misc.health.smithi170.stdout: 2024-03-27T09:34:40.898 INFO:teuthology.misc.health.smithi170.stdout:{"epoch":10,"fsid":"2512214e-2ed3-4e43-8731-25b1a524cd43","created":"2024-03-27T09:34:30.956471+0000","modified":"2024-03-27T09:34:40.721203+0000","last_up_change":"2024-03-27T09:34:38.713503+0000","last_in_change":"2024-03-27T09:34:33.109731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T09:34:39.755285+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"22407295-246e-48a3-976e-795ef8d66974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6805","nonce":2671417311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":2671417311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6807","nonce":2671417311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2671417311}]},"public_addr":"172.21.15.170:6805/2671417311","cluster_addr":"172.21.15.170:6804/2671417311","heartbeat_back_addr":"172.21.15.170:6807/2671417311","heartbeat_front_addr":"172.21.15.170:6808/2671417311","state":["exists","up"]},{"osd":1,"uuid":"bfeb4b6f-ce32-4f7f-b280-31c298b1ced3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":654670318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6801","nonce":654670318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6803","nonce":654670318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":654670318}]},"public_addr":"172.21.15.170:6802/654670318","cluster_addr":"172.21.15.170:6801/654670318","heartbeat_back_addr":"172.21.15.170:6803/654670318","heartbeat_front_addr":"172.21.15.170:6800/654670318","state":["exists","up"]},{"osd":2,"uuid":"e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":1889416444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1889416444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":1889416444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1889416444}]},"public_addr":"172.21.15.172:6805/1889416444","cluster_addr":"172.21.15.172:6804/1889416444","heartbeat_back_addr":"172.21.15.172:6815/1889416444","heartbeat_front_addr":"172.21.15.172:6814/1889416444","state":["exists","up"]},{"osd":3,"uuid":"77bd27c5-42c4-4560-973c-39a14c1d746b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1719380716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1719380716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":1719380716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1719380716}]},"public_addr":"172.21.15.172:6807/1719380716","cluster_addr":"172.21.15.172:6808/1719380716","heartbeat_back_addr":"172.21.15.172:6809/1719380716","heartbeat_front_addr":"172.21.15.172:6806/1719380716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T09:34:40.910 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-03-27T09:34:40.910 INFO:tasks.ceph:Creating RBD pool 2024-03-27T09:34:40.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-27T09:34:41.021 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:41.013+0000 7efc10064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:41.025 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:41.024+0000 7efc10064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:41.724 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'rbd' created 2024-03-27T09:34:41.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-27T09:34:41.876 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:41.865+0000 7fc928cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:41.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:41.880+0000 7fc928cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:42.730 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-27T09:34:42.747 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-27T09:34:42.747 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-27T09:34:42.747 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-27T09:34:42.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-27T09:34:42.748 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-27T09:34:42.908 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:42.904+0000 7f0c3af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:42.988 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:42.987+0000 7f0c3af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:43.232 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:43.244 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":4,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3160268320},{"type":"v1","addr":"172.21.15.172:6801","nonce":3160268320}]},"active_addr":"172.21.15.172:6801/3160268320","active_change":"2024-03-27T09:34:35.705923+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3524564550}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":321019821}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3461222559}]}]} 2024-03-27T09:34:43.247 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-27T09:34:43.247 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T09:34:43.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T09:34:43.363 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:43.358+0000 7f6325c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:43.367 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:43.366+0000 7f6325c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:43.567 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:43.567 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":12,"fsid":"2512214e-2ed3-4e43-8731-25b1a524cd43","created":"2024-03-27T09:34:30.956471+0000","modified":"2024-03-27T09:34:42.728180+0000","last_up_change":"2024-03-27T09:34:38.713503+0000","last_in_change":"2024-03-27T09:34:33.109731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T09:34:39.755285+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T09:34:41.230949+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"22407295-246e-48a3-976e-795ef8d66974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6805","nonce":2671417311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":2671417311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6807","nonce":2671417311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2671417311}]},"public_addr":"172.21.15.170:6805/2671417311","cluster_addr":"172.21.15.170:6804/2671417311","heartbeat_back_addr":"172.21.15.170:6807/2671417311","heartbeat_front_addr":"172.21.15.170:6808/2671417311","state":["exists","up"]},{"osd":1,"uuid":"bfeb4b6f-ce32-4f7f-b280-31c298b1ced3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":654670318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6801","nonce":654670318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6803","nonce":654670318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":654670318}]},"public_addr":"172.21.15.170:6802/654670318","cluster_addr":"172.21.15.170:6801/654670318","heartbeat_back_addr":"172.21.15.170:6803/654670318","heartbeat_front_addr":"172.21.15.170:6800/654670318","state":["exists","up"]},{"osd":2,"uuid":"e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":1889416444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1889416444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":1889416444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1889416444}]},"public_addr":"172.21.15.172:6805/1889416444","cluster_addr":"172.21.15.172:6804/1889416444","heartbeat_back_addr":"172.21.15.172:6815/1889416444","heartbeat_front_addr":"172.21.15.172:6814/1889416444","state":["exists","up"]},{"osd":3,"uuid":"77bd27c5-42c4-4560-973c-39a14c1d746b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1719380716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1719380716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":1719380716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1719380716}]},"public_addr":"172.21.15.172:6807/1719380716","cluster_addr":"172.21.15.172:6808/1719380716","heartbeat_back_addr":"172.21.15.172:6809/1719380716","heartbeat_front_addr":"172.21.15.172:6806/1719380716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T09:34:43.580 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T09:34:43.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T09:34:43.690 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:43.686+0000 7f1571c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:43.694 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:43.693+0000 7f1571c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:43.895 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:43.895 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":13,"fsid":"2512214e-2ed3-4e43-8731-25b1a524cd43","created":"2024-03-27T09:34:30.956471+0000","modified":"2024-03-27T09:34:43.731192+0000","last_up_change":"2024-03-27T09:34:38.713503+0000","last_in_change":"2024-03-27T09:34:33.109731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T09:34:39.755285+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T09:34:41.230949+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"22407295-246e-48a3-976e-795ef8d66974","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6805","nonce":2671417311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":2671417311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6807","nonce":2671417311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2671417311}]},"public_addr":"172.21.15.170:6805/2671417311","cluster_addr":"172.21.15.170:6804/2671417311","heartbeat_back_addr":"172.21.15.170:6807/2671417311","heartbeat_front_addr":"172.21.15.170:6808/2671417311","state":["exists","up"]},{"osd":1,"uuid":"bfeb4b6f-ce32-4f7f-b280-31c298b1ced3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":654670318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6801","nonce":654670318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6803","nonce":654670318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":654670318}]},"public_addr":"172.21.15.170:6802/654670318","cluster_addr":"172.21.15.170:6801/654670318","heartbeat_back_addr":"172.21.15.170:6803/654670318","heartbeat_front_addr":"172.21.15.170:6800/654670318","state":["exists","up"]},{"osd":2,"uuid":"e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":1889416444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1889416444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":1889416444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1889416444}]},"public_addr":"172.21.15.172:6805/1889416444","cluster_addr":"172.21.15.172:6804/1889416444","heartbeat_back_addr":"172.21.15.172:6815/1889416444","heartbeat_front_addr":"172.21.15.172:6814/1889416444","state":["exists","up"]},{"osd":3,"uuid":"77bd27c5-42c4-4560-973c-39a14c1d746b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":1719380716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1719380716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":1719380716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1719380716}]},"public_addr":"172.21.15.172:6807/1719380716","cluster_addr":"172.21.15.172:6808/1719380716","heartbeat_back_addr":"172.21.15.172:6809/1719380716","heartbeat_front_addr":"172.21.15.172:6806/1719380716","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T09:34:43.909 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T09:34:43.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T09:34:43.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T09:34:43.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-27T09:34:44.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.027+0000 7fae9ab54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.027+0000 7fd802d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.027+0000 7f26e6f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.034 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.027+0000 7fd4a95ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.037 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.037+0000 7fae9ab54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.037 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.037+0000 7fd4a3fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.038 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.037+0000 7f26e6f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.038 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.038+0000 7fd802d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.070 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-27T09:34:44.074 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T09:34:44.075 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T09:34:44.076 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T09:34:44.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.180+0000 7f863bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.192 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.192+0000 7f863bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.234 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.229+0000 7f75a440f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.239 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.233+0000 7f017d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.239 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.238+0000 7f75a440f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.241 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.235+0000 7f06f906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.244 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.244+0000 7f017d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.246 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.245+0000 7f06f906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.420 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.3 2024-03-27T09:34:44.434 DEBUG:teuthology.parallel:result is None 2024-03-27T09:34:44.436 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.442 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2024-03-27T09:34:44.451 DEBUG:teuthology.parallel:result is None 2024-03-27T09:34:44.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.2 2024-03-27T09:34:44.457 DEBUG:teuthology.parallel:result is None 2024-03-27T09:34:44.458 INFO:teuthology.orchestra.run.smithi170.stdout:38654705669 2024-03-27T09:34:44.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-03-27T09:34:44.471 DEBUG:teuthology.parallel:result is None 2024-03-27T09:34:44.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T09:34:44.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:34:44.621 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.617+0000 7ff75006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.625 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.624+0000 7ff75006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.815 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:44.815 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:34:44.828 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-27T09:34:43.751054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107920,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379440,"statfs":{"total":386547056640,"available":386436546560,"internally_reserved":0,"allocated":602112,"data_stored":1873234,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.839247+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:34:42.839247+0000","last_peered":"2024-03-27T09:34:42.839247+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:34:42.839247+0000","last_undegraded":"2024-03-27T09:34:42.839247+0000","last_fullsized":"2024-03-27T09:34:42.839247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.846268+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:34:42.846268+0000","last_peered":"2024-03-27T09:34:42.846268+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:34:42.846268+0000","last_undegraded":"2024-03-27T09:34:42.846268+0000","last_fullsized":"2024-03-27T09:34:42.846268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.824140+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:34:42.824140+0000","last_peered":"2024-03-27T09:34:42.824140+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:34:42.824140+0000","last_undegraded":"2024-03-27T09:34:42.824140+0000","last_fullsized":"2024-03-27T09:34:42.824140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.822926+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:34:42.822926+0000","last_peered":"2024-03-27T09:34:42.822926+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:34:42.822926+0000","last_undegraded":"2024-03-27T09:34:42.822926+0000","last_fullsized":"2024-03-27T09:34:42.822926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826211+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:34:42.826211+0000","last_peered":"2024-03-27T09:34:42.826211+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:34:42.826211+0000","last_undegraded":"2024-03-27T09:34:42.826211+0000","last_fullsized":"2024-03-27T09:34:42.826211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.747843+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:34:43.747843+0000","last_peered":"2024-03-27T09:34:43.747843+0000","last_clean":"2024-03-27T09:34:43.747843+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:34:43.747843+0000","last_undegraded":"2024-03-27T09:34:43.747843+0000","last_fullsized":"2024-03-27T09:34:43.747843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826142+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:34:42.826142+0000","last_peered":"2024-03-27T09:34:42.826142+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:34:42.826142+0000","last_undegraded":"2024-03-27T09:34:42.826142+0000","last_fullsized":"2024-03-27T09:34:42.826142+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.849094+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:34:42.849094+0000","last_peered":"2024-03-27T09:34:42.849094+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:34:42.849094+0000","last_undegraded":"2024-03-27T09:34:42.849094+0000","last_fullsized":"2024-03-27T09:34:42.849094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.395934+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:34:43.395934+0000","last_peered":"2024-03-27T09:34:43.395934+0000","last_clean":"2024-03-27T09:34:43.395934+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:34:43.395934+0000","last_undegraded":"2024-03-27T09:34:43.395934+0000","last_fullsized":"2024-03-27T09:34:43.395934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:34:44.829 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:34:44.940 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.936+0000 7fd17ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:44.944 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:44.943+0000 7fd17ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.128 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:45.128 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:34:45.140 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-27T09:34:43.751054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107920,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379440,"statfs":{"total":386547056640,"available":386436546560,"internally_reserved":0,"allocated":602112,"data_stored":1873234,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.839247+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:34:42.839247+0000","last_peered":"2024-03-27T09:34:42.839247+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:34:42.839247+0000","last_undegraded":"2024-03-27T09:34:42.839247+0000","last_fullsized":"2024-03-27T09:34:42.839247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.846268+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:34:42.846268+0000","last_peered":"2024-03-27T09:34:42.846268+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:34:42.846268+0000","last_undegraded":"2024-03-27T09:34:42.846268+0000","last_fullsized":"2024-03-27T09:34:42.846268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.824140+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:34:42.824140+0000","last_peered":"2024-03-27T09:34:42.824140+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:34:42.824140+0000","last_undegraded":"2024-03-27T09:34:42.824140+0000","last_fullsized":"2024-03-27T09:34:42.824140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.822926+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:34:42.822926+0000","last_peered":"2024-03-27T09:34:42.822926+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:34:42.822926+0000","last_undegraded":"2024-03-27T09:34:42.822926+0000","last_fullsized":"2024-03-27T09:34:42.822926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826211+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:34:42.826211+0000","last_peered":"2024-03-27T09:34:42.826211+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:34:42.826211+0000","last_undegraded":"2024-03-27T09:34:42.826211+0000","last_fullsized":"2024-03-27T09:34:42.826211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.747843+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:34:43.747843+0000","last_peered":"2024-03-27T09:34:43.747843+0000","last_clean":"2024-03-27T09:34:43.747843+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:34:43.747843+0000","last_undegraded":"2024-03-27T09:34:43.747843+0000","last_fullsized":"2024-03-27T09:34:43.747843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826142+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:34:42.826142+0000","last_peered":"2024-03-27T09:34:42.826142+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:34:42.826142+0000","last_undegraded":"2024-03-27T09:34:42.826142+0000","last_fullsized":"2024-03-27T09:34:42.826142+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.849094+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:34:42.849094+0000","last_peered":"2024-03-27T09:34:42.849094+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:34:42.849094+0000","last_undegraded":"2024-03-27T09:34:42.849094+0000","last_fullsized":"2024-03-27T09:34:42.849094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.395934+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:34:43.395934+0000","last_peered":"2024-03-27T09:34:43.395934+0000","last_clean":"2024-03-27T09:34:43.395934+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:34:43.395934+0000","last_undegraded":"2024-03-27T09:34:43.395934+0000","last_fullsized":"2024-03-27T09:34:43.395934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:34:45.141 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T09:34:45.141 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-27T09:34:45.141 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-27T09:34:45.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-27T09:34:45.250 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.246+0000 7facd206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.253 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.253+0000 7facd206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.475 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:45.475 INFO:teuthology.orchestra.run.smithi170.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-27T09:34:45.489 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-27T09:34:45.489 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-27T09:34:45.499 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-27T09:34:45.500 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sjust-crimson-testing-2024-03-20'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-thrash-osd', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6db07181a4df6107bd2ae48adf3d2421c02fa2c2'}} 2024-03-27T09:34:45.500 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-27T09:34:45.500 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-27T09:34:45.613 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.605+0000 7f829f417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.616 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.616+0000 7f829f417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.823 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-27T09:34:45.823 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T09:34:45.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:34:45.933 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.929+0000 7f3e7ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:45.937 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:45.936+0000 7f3e7ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:46.120 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:46.120 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:34:46.132 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-27T09:34:45.711212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107920,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379440,"statfs":{"total":386547056640,"available":386436546560,"internally_reserved":0,"allocated":602112,"data_stored":1873234,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960157"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.751972+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:34:43.751972+0000","last_peered":"2024-03-27T09:34:43.751972+0000","last_clean":"2024-03-27T09:34:43.751972+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:34:43.751972+0000","last_undegraded":"2024-03-27T09:34:43.751972+0000","last_fullsized":"2024-03-27T09:34:43.751972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.846268+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:34:42.846268+0000","last_peered":"2024-03-27T09:34:42.846268+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:34:42.846268+0000","last_undegraded":"2024-03-27T09:34:42.846268+0000","last_fullsized":"2024-03-27T09:34:42.846268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.824140+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:34:42.824140+0000","last_peered":"2024-03-27T09:34:42.824140+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:34:42.824140+0000","last_undegraded":"2024-03-27T09:34:42.824140+0000","last_fullsized":"2024-03-27T09:34:42.824140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.822926+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:34:42.822926+0000","last_peered":"2024-03-27T09:34:42.822926+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:34:42.822926+0000","last_undegraded":"2024-03-27T09:34:42.822926+0000","last_fullsized":"2024-03-27T09:34:42.822926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826211+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:34:42.826211+0000","last_peered":"2024-03-27T09:34:42.826211+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:34:42.826211+0000","last_undegraded":"2024-03-27T09:34:42.826211+0000","last_fullsized":"2024-03-27T09:34:42.826211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.747843+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:34:43.747843+0000","last_peered":"2024-03-27T09:34:43.747843+0000","last_clean":"2024-03-27T09:34:43.747843+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:34:43.747843+0000","last_undegraded":"2024-03-27T09:34:43.747843+0000","last_fullsized":"2024-03-27T09:34:43.747843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826142+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:34:42.826142+0000","last_peered":"2024-03-27T09:34:42.826142+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:34:42.826142+0000","last_undegraded":"2024-03-27T09:34:42.826142+0000","last_fullsized":"2024-03-27T09:34:42.826142+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.752926+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:34:43.752926+0000","last_peered":"2024-03-27T09:34:43.752926+0000","last_clean":"2024-03-27T09:34:43.752926+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:34:43.752926+0000","last_undegraded":"2024-03-27T09:34:43.752926+0000","last_fullsized":"2024-03-27T09:34:43.752926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.395934+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:34:43.395934+0000","last_peered":"2024-03-27T09:34:43.395934+0000","last_clean":"2024-03-27T09:34:43.395934+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:34:43.395934+0000","last_undegraded":"2024-03-27T09:34:43.395934+0000","last_fullsized":"2024-03-27T09:34:43.395934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:34:46.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:34:46.243 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:46.239+0000 7f0e9f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:46.247 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:46.246+0000 7f0e9f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:46.432 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:46.432 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:34:46.444 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-27T09:34:45.711212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107920,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379440,"statfs":{"total":386547056640,"available":386436546560,"internally_reserved":0,"allocated":602112,"data_stored":1873234,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960157"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.751972+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:34:43.751972+0000","last_peered":"2024-03-27T09:34:43.751972+0000","last_clean":"2024-03-27T09:34:43.751972+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:34:43.751972+0000","last_undegraded":"2024-03-27T09:34:43.751972+0000","last_fullsized":"2024-03-27T09:34:43.751972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.846268+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:34:42.846268+0000","last_peered":"2024-03-27T09:34:42.846268+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:34:42.846268+0000","last_undegraded":"2024-03-27T09:34:42.846268+0000","last_fullsized":"2024-03-27T09:34:42.846268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.824140+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:34:42.824140+0000","last_peered":"2024-03-27T09:34:42.824140+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:34:42.824140+0000","last_undegraded":"2024-03-27T09:34:42.824140+0000","last_fullsized":"2024-03-27T09:34:42.824140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.822926+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:34:42.822926+0000","last_peered":"2024-03-27T09:34:42.822926+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:34:42.822926+0000","last_undegraded":"2024-03-27T09:34:42.822926+0000","last_fullsized":"2024-03-27T09:34:42.822926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826211+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:34:42.826211+0000","last_peered":"2024-03-27T09:34:42.826211+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:34:42.826211+0000","last_undegraded":"2024-03-27T09:34:42.826211+0000","last_fullsized":"2024-03-27T09:34:42.826211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.747843+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:34:43.747843+0000","last_peered":"2024-03-27T09:34:43.747843+0000","last_clean":"2024-03-27T09:34:43.747843+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:34:43.747843+0000","last_undegraded":"2024-03-27T09:34:43.747843+0000","last_fullsized":"2024-03-27T09:34:43.747843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826142+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:34:42.826142+0000","last_peered":"2024-03-27T09:34:42.826142+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:34:42.826142+0000","last_undegraded":"2024-03-27T09:34:42.826142+0000","last_fullsized":"2024-03-27T09:34:42.826142+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.752926+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:34:43.752926+0000","last_peered":"2024-03-27T09:34:43.752926+0000","last_clean":"2024-03-27T09:34:43.752926+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:34:43.752926+0000","last_undegraded":"2024-03-27T09:34:43.752926+0000","last_fullsized":"2024-03-27T09:34:43.752926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.395934+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:34:43.395934+0000","last_peered":"2024-03-27T09:34:43.395934+0000","last_clean":"2024-03-27T09:34:43.395934+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:34:43.395934+0000","last_undegraded":"2024-03-27T09:34:43.395934+0000","last_fullsized":"2024-03-27T09:34:43.395934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:34:46.445 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T09:34:46.445 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-27T09:34:46.555 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:46.551+0000 7f40f4eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:46.559 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:46.558+0000 7f40f4eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:46.745 INFO:teuthology.orchestra.run.smithi170.stdout:epoch 13 2024-03-27T09:34:46.745 INFO:teuthology.orchestra.run.smithi170.stdout:fsid 2512214e-2ed3-4e43-8731-25b1a524cd43 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:created 2024-03-27T09:34:30.956471+0000 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:modified 2024-03-27T09:34:43.731192+0000 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:crush_version 3 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:full_ratio 0.95 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:backfillfull_ratio 0.9 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:nearfull_ratio 0.85 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:require_min_compat_client luminous 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:min_compat_client jewel 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:require_osd_release squid 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:stretch_mode_enabled false 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:allow_crimson=true 2024-03-27T09:34:46.746 INFO:teuthology.orchestra.run.smithi170.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:max_osd 4 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.170:6805/2671417311 v2:172.21.15.170:6804/2671417311 exists,up 22407295-246e-48a3-976e-795ef8d66974 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.170:6802/654670318 v2:172.21.15.170:6801/654670318 exists,up bfeb4b6f-ce32-4f7f-b280-31c298b1ced3 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6805/1889416444 v2:172.21.15.172:6804/1889416444 exists,up e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1 2024-03-27T09:34:46.747 INFO:teuthology.orchestra.run.smithi170.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6807/1719380716 v2:172.21.15.172:6808/1719380716 exists,up 77bd27c5-42c4-4560-973c-39a14c1d746b 2024-03-27T09:34:46.758 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.170:6805/2671417311 v2:172.21.15.170:6804/2671417311 exists,up 22407295-246e-48a3-976e-795ef8d66974', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.170:6802/654670318 v2:172.21.15.170:6801/654670318 exists,up bfeb4b6f-ce32-4f7f-b280-31c298b1ced3', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6805/1889416444 v2:172.21.15.172:6804/1889416444 exists,up e8ec2350-5c9f-4518-83f1-9d5f8ddc23c1', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.172:6807/1719380716 v2:172.21.15.172:6808/1719380716 exists,up 77bd27c5-42c4-4560-973c-39a14c1d746b'] 2024-03-27T09:34:46.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "name": "mon.a", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "cluster": "ceph", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "admin_socket_mode": "", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_client_required": "cephx, none", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_cluster_required": "cephx", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_debug": "true", 2024-03-27T09:34:46.867 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_service_required": "cephx", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "auth_supported": "cephx", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_aio": "true", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_aio_poll_ms": "250", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_aio_reap_max": "16", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_async_discard_threads": "0", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_block_size": "4096", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_debug_aio": "false", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-27T09:34:46.868 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_debug_inflight_ios": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_enable_discard": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_flock_retry": "3", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_inject_crash": "0", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_ioring": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_ioring_hipri": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bdev_type": "", 2024-03-27T09:34:46.869 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_alloc_size": "1048576", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_allocator": "hybrid", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_buffered_io": "true", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_check_for_zeros": "false", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_compact_log_sync": "false", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_max_log_runway": "4194304", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_max_prefetch": "1048576", 2024-03-27T09:34:46.870 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_min_flush_size": "524288", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_min_log_runway": "1048576", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_replay_recovery": "false", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_sync_write": "false", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_allocation_from_file": "true", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_allocator": "hybrid", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-27T09:34:46.871 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_create": "true", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_db_create": "false", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_db_path": "", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_db_size": "0", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_path": "", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_preallocate_file": "false", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_size": "107374182400", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_wal_create": "false", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_wal_path": "", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_block_wal_size": "100663296", 2024-03-27T09:34:46.872 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bluefs": "true", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_autotune": "true", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-27T09:34:46.873 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_size": "0", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_type": "2q", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_clone_cow": "true", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-27T09:34:46.874 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_mode": "none", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_csum_type": "crc32c", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_freelist": "false", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-27T09:34:46.875 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_small_allocations": "0", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_default_buffered_read": "true", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_default_buffered_write": "false", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-27T09:34:46.876 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fail_eio": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_mount": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-27T09:34:46.877 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_umount": "false", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_ignore_data_csum": "false", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-27T09:34:46.878 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_log_op_age": "5.000000", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_alloc_size": "0", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_blob_size": "0", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_max_deferred_txc": "32", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_min_alloc_size": "0", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_nid_prealloc": "1024", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-27T09:34:46.879 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_retry_disk_reads": "3", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_rocksdb_cf": "true", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_spdk_mem": "512", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-27T09:34:46.880 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_tracing": "false", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_zero_block_detection": "false", 2024-03-27T09:34:46.881 INFO:teuthology.orchestra.run.smithi170.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-27T09:34:46.882 INFO:teuthology.orchestra.run.smithi170.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_cluster_require_signatures": "false", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_cluster_require_version": "2", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_require_signatures": "false", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_require_version": "2", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_service_require_signatures": "false", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_service_require_version": "2", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "cephx_sign_messages": "true", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "chdir": "", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "client_acl_type": "", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "client_asio_thread_count": "2", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "client_cache_mid": "0.750000", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "client_cache_size": "16384", 2024-03-27T09:34:46.883 INFO:teuthology.orchestra.run.smithi170.stdout: "client_caps_release_delay": "5", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_check_pool_perm": "true", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_debug_force_sync_read": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_debug_getattr_caps": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_debug_inject_tick_delay": "0", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_die_on_failed_remount": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_dirsize_rbytes": "true", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_force_lazyio": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_fs": "", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_inject_release_failure": "false", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_max_inline_size": "4096", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-27T09:34:46.884 INFO:teuthology.orchestra.run.smithi170.stdout: "client_mds_namespace": "", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_metadata": "", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_mount_gid": "-1", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_mount_timeout": "300", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_mount_uid": "-1", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_mountpoint": "/", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_notify_timeout": "10", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc": "true", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc_max_dirty": "104857600", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc_max_objects": "1000", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc_size": "209715200", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_oc_target_dirty": "8388608", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_permissions": "true", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_quota": "true", 2024-03-27T09:34:46.885 INFO:teuthology.orchestra.run.smithi170.stdout: "client_quota_df": "true", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_readahead_max_bytes": "0", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_readahead_max_periods": "4", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_readahead_min": "131072", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_reconnect_stale": "false", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_shutdown_timeout": "30", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_snapdir": ".snap", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_tick_interval": "1", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_trace": "", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_try_dentry_invalidate": "false", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_use_faked_inos": "false", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_use_random_mds": "false", 2024-03-27T09:34:46.886 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_graylog": "false", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_graylog_port": "12201", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_monitors": "default=true", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_syslog": "false", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "clog_to_syslog_level": "info", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "cluster_addr": "-", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "cluster_network": "", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "cluster_network_interface": "", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "compressor_zlib_isal": "false", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "compressor_zlib_level": "5", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "compressor_zlib_winsize": "-15", 2024-03-27T09:34:46.887 INFO:teuthology.orchestra.run.smithi170.stdout: "compressor_zstd_level": "1", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crimson_alien_op_num_threads": "6", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crimson_seastar_cpu_cores": "", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crush_location": "", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crush_location_hook": "", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "crush_location_hook_timeout": "10", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "daemonize": "false", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "daos_pool": "tank", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-27T09:34:46.888 INFO:teuthology.orchestra.run.smithi170.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_alienstore": "0/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_allow_any_pool_priority": "false", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_asok": "1/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_asok_assert_abort": "false", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_asserts_on_shutdown": "true", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_auth": "20/20", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_bdev": "1/3", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_bluefs": "1/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_bluestore": "1/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_buffer": "0/1", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_ceph_exporter": "1/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_cephfs_mirror": "0/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_cephsqlite": "0/5", 2024-03-27T09:34:46.889 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_client": "0/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_compressor": "1/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_context": "0/1", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_crimson_interrupt": "0/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_crush": "1/1", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_crypto": "1/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_cyanstore": "0/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_deliberately_leak_memory": "false", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_disable_randomized_ping": "false", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_dpdk": "1/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_eventtrace": "1/5", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_filer": "0/1", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_filestore": "1/3", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_finisher": "1/1", 2024-03-27T09:34:46.890 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_fuse": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_heartbeat_testing_span": "0", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_heartbeatmap": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_javaclient": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_journal": "1/3", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_journaler": "0/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_kstore": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_lockdep": "0/1", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mclock": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_balancer": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_locker": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_log": "1/5", 2024-03-27T09:34:46.891 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_log_expire": "1/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_migrator": "1/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mds_quiesce": "3/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_memstore": "1/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mgr": "2/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mgrc": "1/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_mon": "20/20", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_monc": "0/10", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_ms": "1/1", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_none": "0/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_objclass": "0/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_objectcacher": "0/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_objecter": "0/1", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_optracker": "0/5", 2024-03-27T09:34:46.892 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_osd": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_paxos": "20/20", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_perfcounter": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_prioritycache": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rados": "0/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rbd": "0/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rbd_mirror": "0/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rbd_pwl": "0/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rbd_replay": "0/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_refs": "0/0", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_reserver": "1/1", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_access": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_datacache": "1/5", 2024-03-27T09:34:46.893 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_dbstore": "1/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_flight": "1/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rgw_sync": "1/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_rocksdb": "4/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_backref": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_cache": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_cleaner": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_device": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_epm": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_journal": "0/5", 2024-03-27T09:34:46.894 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_lba": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_odata": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_omap": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_onode": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_t": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_seastore_tm": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_striper": "0/1", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_test": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_throttle": "1/1", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_timer": "0/1", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_tp": "0/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "debug_trace": "1/5", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "device_failure_prediction_mode": "none", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-03-27T09:34:46.895 INFO:teuthology.orchestra.run.smithi170.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "err_to_graylog": "false", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "err_to_journald": "false", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "err_to_stderr": "true", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "err_to_syslog": "false", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "event_tracing": "false", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_addr": "0.0.0.0", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_http_port": "9926", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_prio_limit": "5", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_sort_metrics": "true", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "exporter_stats_period": "5", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "fake_statfs_for_testing": "0", 2024-03-27T09:34:46.896 INFO:teuthology.orchestra.run.smithi170.stdout: "fatal_signal_handlers": "true", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filer_max_purge_ops": "10", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filer_max_truncate_ops": "128", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_apply_finisher_threads": "1", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_blackhole": "false", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_btrfs_clone_range": "true", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_btrfs_snap": "true", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_caller_concurrency": "10", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_collect_device_partition_information": "true", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_commit_timeout": "600.000000", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_debug_inject_read_err": "false", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_debug_omap_check": "false", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_debug_verify_split": "false", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_dump_file": "", 2024-03-27T09:34:46.897 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fadvise": "true", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fail_eio": "true", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fd_cache_shards": "16", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fd_cache_size": "128", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fiemap": "false", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fiemap_threshold": "4096", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_inject_stall": "0", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_journal_parallel": "false", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_journal_trailing": "false", 2024-03-27T09:34:46.898 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_journal_writeahead": "false", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_kill_at": "0", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattrs": "0", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_xattr_value_size": "0", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-27T09:34:46.899 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_merge_threshold": "-10", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_odsync_write": "false", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_omap_backend": "rocksdb", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_omap_backend_path": "", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_op_thread_timeout": "60", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_op_threads": "2", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_punch_hole": "false", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-27T09:34:46.900 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_queue_max_ops": "50", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_seek_data_hole": "false", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_sloppy_crc": "false", 2024-03-27T09:34:46.901 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_splice": "false", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_split_multiple": "2", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_split_rand_factor": "20", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_update_to": "1000", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_enable": "true", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-27T09:34:46.902 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_xfs_extsize": "false", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "filestore_zfs_snap": "false", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fio_dir": "/tmp/fio", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fsid": "2512214e-2ed3-4e43-8731-25b1a524cd43", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_allow_other": "true", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_atomic_o_trunc": "true", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_big_writes": "true", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_debug": "false", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_default_permissions": "false", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_disable_pagecache": "false", 2024-03-27T09:34:46.903 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_max_write": "0", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_multithreaded": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_require_active_mds": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_set_user_groups": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_splice_move": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_splice_read": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_splice_write": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "fuse_use_invalidate_cb": "true", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "gss_target_name": "ceph", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "heartbeat_file": "", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "heartbeat_inject_failure": "0", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "heartbeat_interval": "5", 2024-03-27T09:34:46.904 INFO:teuthology.orchestra.run.smithi170.stdout: "host": "smithi170", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-27T09:34:46.905 INFO:teuthology.orchestra.run.smithi170.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "inject_early_sigterm": "false", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "jaeger_agent_port": "6799", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "jaeger_tracing_enable": "false", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_aio": "true", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_align_min_size": "65536", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_block_align": "true", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_block_size": "4096", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_dio": "true", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_discard": "false", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_force_aio": "false", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_ignore_corruption": "false", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_max_write_bytes": "10485760", 2024-03-27T09:34:46.906 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_max_write_entries": "100", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_replay_from": "0", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_write_header_frequency": "0", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journal_zero_on_create": "false", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journaler_prefetch_periods": "10", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journaler_prezero_periods": "5", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "journaler_write_head_interval": "15", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "key": "", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "keyfile": "", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_backend": "rocksdb", 2024-03-27T09:34:46.907 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_default_stripe_size": "65536", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_fsck_on_mount": "false", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_max_bytes": "67108864", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_max_ops": "512", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_nid_prealloc": "1024", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_onode_map_size": "1024", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_sync_submit_transaction": "false", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "kstore_sync_transaction": "false", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "librados_thread_count": "2", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "lockdep": "false", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "lockdep_force_backtrace": "false", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "log_coarse_timestamps": "true", 2024-03-27T09:34:46.908 INFO:teuthology.orchestra.run.smithi170.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_flush_on_exit": "false", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_graylog_host": "127.0.0.1", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_graylog_port": "12201", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_max_new": "1000", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_max_recent": "10000", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_stderr_prefix": "", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_stop_at_utilization": "0.970000", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_to_file": "true", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_to_graylog": "false", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_to_journald": "false", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_to_stderr": "false", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "log_to_syslog": "false", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "max_rotating_auth_attempts": "10", 2024-03-27T09:34:46.909 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_action_on_write_error": "1", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_alternate_name_max": "8192", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_asio_thread_count": "2", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_export_pin": "true", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_fragment_dirs": "true", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_fragment_interval": "5", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_interval": "10", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_max": "-1", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_max_until": "-1", 2024-03-27T09:34:46.910 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_merge_size": "50", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_midchunk": "0.300000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_min_start": "0.200000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_minchunk": "0.001000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_mode": "0", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_need_max": "1.200000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_need_min": "0.800000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_overload_epochs": "2", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_split_bits": "3", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_split_size": "10000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-27T09:34:46.911 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_target_decay": "10.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_beacon_grace": "15.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_beacon_interval": "4.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_mid": "0.700000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_release_free_interval": "10", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_reservation": "0.050000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_trim_interval": "1", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache_trim_threshold": "262144", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-27T09:34:46.912 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_client_prealloc_inos": "1000", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_connect_bootstrapping": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_damage_table_max_entries": "10000", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_debug_auth_pins": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_debug_frag": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_debug_scatterstat": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_debug_subtrees": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_decay_halflife": "5.000000", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_default_dir_hash": "2", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_defer_session_stale": "true", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_deny_all_reconnect": "false", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dir_keys_per_op": "16384", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dir_max_commit_size": "10", 2024-03-27T09:34:46.913 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dir_max_entries": "0", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dir_prefetch": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dump_cache_on_map": "false", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_early_reply": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_enable_op_tracker": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_enforce_unique_name": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_export_ephemeral_random": "true", 2024-03-27T09:34:46.914 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_health_summarize_threshold": "10", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_health_dummy": "false", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_migrator_session_race": "false", 2024-03-27T09:34:46.915 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_join_fs": "", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_journal_format": "1", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_create_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_export_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_import_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_journal_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_journal_expire_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_journal_replay_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_link_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_mdstable_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_openc_at": "0", 2024-03-27T09:34:46.916 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_rename_at": "0", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_shutdown_at": "0", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_event_large_threshold": "524288", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_events_per_segment": "1024", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_max_events": "-1", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_max_segments": "128", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_pause": "false", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_segment_size": "0", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_trim_threshold": "128", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-27T09:34:46.917 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log_warn_factor": "2.000000", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_caps_per_client": "1048576", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_completed_flushes": "100000", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_completed_requests": "100000", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_export_size": "20971520", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_file_recover": "32", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_purge_files": "64", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_purge_ops": "8192", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_max_snaps_per_dir": "100", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_metrics_update_interval": "2", 2024-03-27T09:34:46.918 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_min_caps_per_client": "100", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_min_caps_working_set": "10000", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_numa_node": "-1", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_complaint_time": "30.000000", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_history_duration": "600", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_history_size": "20", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_history_slow_op_size": "20", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_op_log_threshold": "5", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_ping_grace": "15", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_ping_interval": "5", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-27T09:34:46.919 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_max_caps": "30000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_recall_warning_threshold": "262144", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_replay_interval": "1.000000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_root_ino_gid": "0", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_root_ino_uid": "0", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-27T09:34:46.920 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_shutdown_check": "0", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_skip_ino": "0", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_snap_max_uid": "4294967294", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_snap_min_uid": "0", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_snap_rstat": "false", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_standby_replay_damaged": "false", 2024-03-27T09:34:46.921 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_symlink_recovery": "true", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_thrash_exports": "0", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_thrash_fragments": "0", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_tick_interval": "5.000000", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_valgrind_exit": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_verify_backtrace": "1", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_verify_scatter": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_wipe_sessions": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool_debug": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "memstore_device_bytes": "1073741824", 2024-03-27T09:34:46.922 INFO:teuthology.orchestra.run.smithi170.stdout: "memstore_page_set": "false", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "memstore_page_size": "65536", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_client_bytes": "134217728", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_client_messages": "512", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_disabled_modules": "", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_max_pg_num_change": "128", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_mds_bytes": "134217728", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_mds_messages": "128", 2024-03-27T09:34:46.923 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_mon_bytes": "134217728", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_mon_messages": "128", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_osd_bytes": "536870912", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_osd_messages": "8192", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_pool": "true", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_standby_modules": "true", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_stats_period": "5", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_stats_threshold": "5", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_tick_period": "2", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-27T09:34:46.924 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_allow_pool_delete": "true", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_allow_pool_size_one": "true", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_auth_validate_all_caps": "true", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_bytes": "104857600", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_directed_command_retry": "2", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_hunt_parallel": "3", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_log_interval": "1.000000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-27T09:34:46.925 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_ping_interval": "10.000000", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_client_target_rank": "-1", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_file_level": "debug", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_file": "true", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_journald": "false", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-27T09:34:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_compact_on_bootstrap": "false", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_compact_on_start": "false", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_compact_on_trim": "true", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_cpu_threads": "4", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_crush_min_required_version": "hammer", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_daemon_bytes": "419430400", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_data_avail_crit": "5", 2024-03-27T09:34:46.927 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_data_avail_warn": "5", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_data_size_warn": "16106127360", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_dump_json": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_dump_transactions": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_extra_checks": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_no_require_reef": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_no_require_squid": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-27T09:34:46.928 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_down_mkfs_grace": "120", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_election_default_strategy": "1", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_election_timeout": "5.000000", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_elector_ping_divisor": "2", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_enable_op_tracker": "true", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_fake_pool_delete": "false", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_force_quorum_join": "false", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_globalid_prealloc": "10000", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_detail_to_clog": "false", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_log_update_period": "5", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_max_detail": "50", 2024-03-27T09:34:46.929 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_to_clog": "true", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_to_clog_interval": "600", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_host": "172.21.15.170", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_host_override": "", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_initial_members": "", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_lease": "5.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-27T09:34:46.930 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_log_full_interval": "50", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_log_max": "10000", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_log_max_summary": "50", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_log_epochs": "500", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_osd": "10000", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_pg_per_osd": "10000", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_pool_pg_num": "65536", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mds_force_trim_to": "0", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mds_skip_sanity": "false", 2024-03-27T09:34:46.931 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_memory_autotune": "true", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_memory_target": "2147483648", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_beacon_grace": "30", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_digest_period": "5", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_inactive_grace": "60", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_min_osdmap_epochs": "50", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_complaint_time": "30", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_history_duration": "600", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_history_size": "20", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_history_slow_op_size": "20", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_op_log_threshold": "5", 2024-03-27T09:34:46.932 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_auto_mark_in": "false", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_cache_size": "500", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_down_out_interval": "600", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_force_trim_to": "0", 2024-03-27T09:34:46.933 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_min_down_reporters": "2", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-27T09:34:46.934 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_report_timeout": "900", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_stuck_threshold": "60", 2024-03-27T09:34:46.935 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_probe_timeout": "2.000000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_reweight_max_change": "0.050000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_reweight_max_osds": "4", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-27T09:34:46.936 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_scrub_interval": "86400", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_scrub_max_keys": "100", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_scrub_timeout": "300", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_session_timeout": "300", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_smart_report_timeout": "5", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_stat_smooth_intervals": "6", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_stretch_pool_min_size": "2", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_stretch_pool_size": "4", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_debug": "false", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_provider_kill_at": "0", 2024-03-27T09:34:46.937 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_requester_kill_at": "0", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_sync_timeout": "60.000000", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_target_pg_per_osd": "100", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_tick_interval": "5", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_timecheck_interval": "300.000000", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_use_min_delay_socket": "false", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_older_version_delay": "604800", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-27T09:34:46.938 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_misplaced": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_older_version": "true", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-27T09:34:46.939 INFO:teuthology.orchestra.run.smithi170.stdout: "monmap": "", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "motr_tracing_enabled": "false", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_op_threads": "3", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_cm": "false", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_device_name": "", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_dscp": "96", 2024-03-27T09:34:46.940 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_local_gid": "", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_port_num": "1", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_sl": "3", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_support_srq": "true", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_rdma_type": "ib", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_async_reap_threshold": "5", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_before_connect": "false", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_ipv4": "true", 2024-03-27T09:34:46.941 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_ipv6": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_msgr1": "true", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_msgr2": "true", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_port_max": "7568", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_port_min": "6800", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_retry_count": "3", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_bind_retry_delay": "5", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_blackhole_client": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_blackhole_mds": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_blackhole_mgr": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_blackhole_mon": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_blackhole_osd": "false", 2024-03-27T09:34:46.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_client_mode": "crc", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_cluster_mode": "crc", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_cluster_type": "", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_compress_secure": "false", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_connection_idle_timeout": "900", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_connection_ready_timeout": "10", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_crc_data": "true", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_crc_header": "true", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_die_on_bad_msg": "false", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_die_on_bug": "true", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_die_on_old_message": "true", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_die_on_skipped_message": "false", 2024-03-27T09:34:46.943 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_coremask": "0xF", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_enable_tso": "true", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_hugepages": "", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_lro": "true", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_memory_channel": "4", 2024-03-27T09:34:46.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_pmd": "", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_port_id": "0", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_dump_on_send": "false", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_initial_backoff": "0.200000", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_delay_max": "1.000000", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_delay_type": "", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_network_congestion": "0", 2024-03-27T09:34:46.945 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_inject_socket_failures": "0", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_learn_addr_from_peer": "true", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_max_accept_failures": "4", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_max_backoff": "15.000000", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_mon_client_mode": "crc", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_mon_cluster_mode": "crc", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_mon_service_mode": "crc", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_osd_compress_min_size": "1024", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_osd_compress_mode": "none", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_pq_min_cost": "65536", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_public_type": "", 2024-03-27T09:34:46.946 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_service_mode": "crc", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_tcp_listen_backlog": "512", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_tcp_nodelay": "true", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_tcp_rcvbuf": "0", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "ms_type": "async+posix", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "no_config_file": "false", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_completion_locks_per_session": "32", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_inflight_ops": "1024", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_tick_interval": "5.000000", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter_timeout": "10.000000", 2024-03-27T09:34:46.947 INFO:teuthology.orchestra.run.smithi170.stdout: "objectstore_blackhole": "false", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "openssl_engine_opts": "", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_delay_time": "5.000000", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_hist_halflife": "1000", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_max_low_ops": "2", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_max_ops": "4", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_agent_slop": "0.020000", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_asio_thread_count": "2", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_async_recovery_min_cost": "100", 2024-03-27T09:34:46.948 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backfill_scan_max": "512", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backfill_scan_min": "64", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backoff_on_degraded": "false", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backoff_on_peering": "false", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_backoff_on_unfound": "true", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_beacon_report_interval": "300", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bench_duration": "30", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bench_max_block_size": "67108864", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_blkin_trace_all": "false", 2024-03-27T09:34:46.949 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_check_for_log_corruption": "false", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_class_update_on_start": "true", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_client_message_cap": "256", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_client_message_size_cap": "524288000", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_client_op_priority": "63", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_client_watch_timeout": "30", 2024-03-27T09:34:46.950 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_command_max_records": "256", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_compact_on_start": "false", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_crush_update_on_start": "true", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_crush_update_weight_set": "true", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_feed_pullee": "-1", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-27T09:34:46.951 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_misdirected_ops": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_no_acting_change": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_no_purge_strays": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_op_order": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_shutdown": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_trim_objects": "false", 2024-03-27T09:34:46.952 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_verify_snaps": "false", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_keys": "1024", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_stride": "524288", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_default_notify_timeout": "30", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_delete_sleep": "0.000000", 2024-03-27T09:34:46.953 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_discard_disconnected_ops": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_enable_op_tracker": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_fast_info": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_fast_shutdown": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-27T09:34:46.954 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_function_tracing": "false", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_grace": "20", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_interval": "6", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_min_peers": "10", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_min_size": "2000", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_stale": "600", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_hit_set_max_size": "100000", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_hit_set_min_size": "1000", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-27T09:34:46.955 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_journal_size": "5120", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_kill_backfill_at": "0", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_map_cache_size": "50", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_map_dedup": "true", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_map_message_max": "40", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_map_share_max_epochs": "40", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_attr_name_len": "100", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_attr_size": "0", 2024-03-27T09:34:46.956 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_backfills": "1", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_markdown_count": "5", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_markdown_period": "600", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_object_name_len": "2048", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_object_namespace_len": "256", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_object_size": "134217728", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_pg_blocked_by": "16", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_pg_log_entries": "10000", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_pgls": "1024", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_push_cost": "8388608", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_push_objects": "10", 2024-03-27T09:34:46.957 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_scrubs": "3", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_trimming_pgs": "2", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_write_op_reply_len": "64", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_max_write_size": "90", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_profile": "balanced", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-27T09:34:46.958 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_base": "805306368", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_cache_min": "134217728", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-27T09:34:46.959 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_target": "4294967296", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_target_autotune": "false", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_min_pg_log_entries": "250", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_min_recovery_priority": "0", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mon_report_interval": "5", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_num_cache_shards": "32", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_num_op_tracker_shard": "32", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_numa_auto_affinity": "true", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_numa_node": "-1", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_numa_prefer_iface": "true", 2024-03-27T09:34:46.960 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_objecter_finishers": "1", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_objectstore": "bluestore", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_objectstore_fuse": "false", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_objectstore_tracing": "false", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_complaint_time": "30.000000", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_history_duration": "600", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_history_size": "20", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_history_slow_op_size": "20", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_log_threshold": "5", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_shards": "0", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_shards_hdd": "5", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_shards_ssd": "8", 2024-03-27T09:34:46.961 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_pq_min_cost": "65536", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_queue_cut_off": "high", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_op_thread_timeout": "15", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_open_classes_on_start": "true", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_os_flags": "0", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_peering_op_priority": "255", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_delete_cost": "1048576", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_delete_priority": "5", 2024-03-27T09:34:46.962 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_log_trim_max": "10000", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_log_trim_min": "100", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-27T09:34:46.963 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_crimson": "true", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_flags": "0", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_min_size": "2", 2024-03-27T09:34:46.964 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_pg_num": "32", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_pgp_num": "0", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_read_ratio": "70", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_size": "3", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_default_type": "replicated", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_push_per_object_cost": "1000", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recover_clone_overlap": "true", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_cost": "20971520", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-27T09:34:46.965 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_active": "0", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_max_single_start": "1", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_op_priority": "3", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_priority": "5", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_sleep": "0.000000", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-27T09:34:46.966 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_repair_during_recovery": "false", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_requested_scrub_priority": "120", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_auto_repair": "false", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_begin_hour": "0", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_begin_week_day": "0", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_chunk_max": "25", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_chunk_min": "5", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_cost": "52428800", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_during_recovery": "false", 2024-03-27T09:34:46.967 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_end_hour": "0", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_end_week_day": "0", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_event_cost": "4096", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_invalid_stats": "true", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_max_preemptions": "5", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_priority": "5", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_sleep": "0.000000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-27T09:34:46.968 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_skip_data_digest": "false", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_smart_report_timeout": "5", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_cost": "1048576", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_priority": "5", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-27T09:34:46.969 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_target_transaction_size": "30", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_tracing": "false", 2024-03-27T09:34:46.970 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_use_stale_snap": "false", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "osdc_blkin_trace_all": "false", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_kill_at": "0", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_max_join_drift": "10", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_min": "500", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_min_wait": "0.050000", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_propose_interval": "1.000000", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_service_trim_max": "500", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_service_trim_min": "10", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_stash_full_interval": "25", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_trim_max": "500", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "paxos_trim_min": "250", 2024-03-27T09:34:46.971 INFO:teuthology.orchestra.run.smithi170.stdout: "perf": "true", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "public_addr": "-", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "public_addrv": "", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "public_bind_addr": "-", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "public_network": "", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "public_network_interface": "", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "qat_compressor_enabled": "false", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "qat_compressor_session_max_number": "256", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "rados_mon_op_timeout": "0", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "rados_osd_op_timeout": "0", 2024-03-27T09:34:46.972 INFO:teuthology.orchestra.run.smithi170.stdout: "rados_tracing": "false", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_atime_update_interval": "60", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_balance_parent_reads": "false", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_balance_snap_reads": "false", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_blkin_trace_all": "false", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache": "true", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_policy": "writearound", 2024-03-27T09:34:46.973 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_size": "33554432", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_clone_copy_on_read": "false", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_compression_hint": "none", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_concurrent_management_ops": "10", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_clone_format": "auto", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_data_pool": "", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_features": "61", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_format": "2", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_map_options": "", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_order": "22", 2024-03-27T09:34:46.974 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_pool": "rbd", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_stripe_count": "0", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_default_stripe_unit": "0", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_enable_alloc_hint": "true", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_io_scheduler": "simple", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-27T09:34:46.975 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_order": "24", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_pool": "", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_journal_splay_width": "4", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_localize_parent_reads": "false", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_localize_snap_reads": "false", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-27T09:34:46.976 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-27T09:34:46.977 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-27T09:34:46.978 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_mtime_update_interval": "60", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_non_blocking_aio": "true", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_op_thread_timeout": "60", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_op_threads": "1", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_parent_cache_enabled": "false", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_plugins": "", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_bps_burst": "0", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_bps_limit": "0", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_exclude_ops": "0", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_iops_burst": "0", 2024-03-27T09:34:46.979 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_iops_limit": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-27T09:34:46.980 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_read_from_replica_policy": "default", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_skip_partial_discard": "true", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_tracing": "false", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_validate_names": "true", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "rbd_validate_pool": "true", 2024-03-27T09:34:46.981 INFO:teuthology.orchestra.run.smithi170.stdout: "restapi_base_url": "", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "restapi_log_level": "", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_acl_grants_max_num": "100", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_admin_entry": "admin", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_backend_store": "rados", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_barbican_url": "", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_beast_enable_async": "true", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_counters_cache": "false", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-27T09:34:46.982 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_cache_enabled": "true", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_cache_expiry_interval": "900", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_cache_lru_size": "10000", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_config_store": "rados", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_content_length_compat": "false", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_copy_obj_progress": "true", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_cors_rules_max_num": "100", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_cross_domain_policy": "", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-27T09:34:46.983 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_addr": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_password": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_kmip_username": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_require_ssl": "true", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-27T09:34:46.984 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_addr": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_auth": "token", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_namespace": "", 2024-03-27T09:34:46.985 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_prefix": "", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_token_file": "", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_curl_buffersize": "524288", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_curl_low_speed_time": "300", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-27T09:34:46.986 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_d4n_port": "6379", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_log_changes_size": "1000", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_log_num_shards": "128", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_log_window": "30", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-27T09:34:46.987 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-27T09:34:46.988 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dns_name": "", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dns_s3website_name": "", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_dynamic_resharding": "true", 2024-03-27T09:34:46.989 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_gc_threads": "true", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_lc_threads": "true", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_ops_log": "false", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_quota_threads": "true", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_static_website": "false", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enable_usage_log": "false", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_enforce_swift_acls": "true", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_exit_timeout_secs": "120", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_expose_bucket": "false", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_extended_http_attrs": "", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_filter": "none", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_frontends": "beast port=7480", 2024-03-27T09:34:46.990 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_deferred": "50", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_objs": "32", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_gc_processor_period": "3600", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_init_timeout": "300", 2024-03-27T09:34:46.991 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_kafka_connection_idle": "300", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_domain": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_password": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_password_path": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_project": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_tenant": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_token": "", 2024-03-27T09:34:46.992 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_token_path": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_admin_user": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_api_version": "2", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_barbican_domain": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_barbican_password": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_barbican_project": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_barbican_user": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_url": "", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-27T09:34:46.993 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_debug_interval": "-1", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_lock_max_time": "90", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_max_objs": "32", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_max_rules": "1000", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_max_worker": "3", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lc_thread_delay": "0", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_dnattr": "uid", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_searchfilter": "", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-27T09:34:46.994 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_log_http_headers": "", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_log_object_name_utc": "false", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_attr_name_len": "0", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_attr_size": "0", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-27T09:34:46.995 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_chunk_size": "4194304", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_listing_results": "1000", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_notify_retries": "10", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_put_param_size": "1048576", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_put_size": "5368709120", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_max_slo_entries": "1000", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_md_log_max_shards": "64", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-27T09:34:46.996 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_mp_lock_max_time": "600", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_max_gc": "300", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-27T09:34:46.997 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_num_async_rados_threads": "32", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_num_control_oids": "8", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_numa_node": "-1", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_objexp_chunk_size": "100", 2024-03-27T09:34:46.998 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_objexp_gc_interval": "600", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_op_thread_timeout": "600", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_op_tracing": "false", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_opa_token": "", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_opa_url": "", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_opa_verify_ssl": "true", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ops_log_rados": "false", 2024-03-27T09:34:46.999 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_ops_log_socket_path": "", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_period_push_interval": "2.000000", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_cache_lanes": "3", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_cache_partitions": "3", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-27T09:34:47.000 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_print_continue": "true", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_rados_tracing": "false", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_realm": "", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_realm_id": "", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_region": "", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-27T09:34:47.001 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_request_uri": "", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_reshard_batch_size": "64", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_reshard_max_aio": "128", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_reshard_num_logs": "16", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_reshard_thread_interval": "600", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_resolve_cname": "false", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_run_sync_thread": "true", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-27T09:34:47.002 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_scheduler_type": "throttler", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_script_uri": "", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_service_provider_name": "", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_client_id": "", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_client_secret": "", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_entry": "sts", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_key": "sts", 2024-03-27T09:34:47.003 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_min_session_duration": "900", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sts_token_introspection_url": "", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_account_in_url": "false", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_auth_entry": "auth", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_auth_url": "", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_custom_header": "", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_need_stats": "true", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_tenant_name": "", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_token_expiration": "86400", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_url": "", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_url_prefix": "swift", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-27T09:34:47.004 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_lease_period": "120", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_thread_pool_size": "512", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-27T09:34:47.005 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_comment": "", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_createby": "", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_encoding": "", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_flag": "false", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_origin": "", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_torrent_tracker": "", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_trust_forwarded_https": "false", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-27T09:34:47.006 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_usage_max_shards": "32", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_usage_max_user_shards": "1", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_use_opa_authz": "false", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_counters_cache": "false", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_max_buckets": "1000", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_policies_max_num": "100", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_user_unique_email": "true", 2024-03-27T09:34:47.007 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_verify_ssl": "true", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zone": "", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zone_id": "", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zonegroup": "", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zonegroup_id": "", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgwlc_auto_session_clear": "true", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_block_size": "4096", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-27T09:34:47.008 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_size": "536870912", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_index_type": "binary_search", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-27T09:34:47.009 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_partition_filters": "false", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_perf": "false", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "run_dir": "/var/run/ceph", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_block_create": "true", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_cache_lru_size": "67108864", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_default_max_object_size": "16777216", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_device_size": "53687091200", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_journal_batch_capacity": "16", 2024-03-27T09:34:47.010 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_main_device_type": "SSD", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "seastore_segment_size": "67108864", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "set_keepcaps": "false", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "setgroup": "", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "setuser": "", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "setuser_match_path": "", 2024-03-27T09:34:47.011 INFO:teuthology.orchestra.run.smithi170.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-27T09:34:47.012 INFO:teuthology.orchestra.run.smithi170.stdout: "thp": "false", 2024-03-27T09:34:47.012 INFO:teuthology.orchestra.run.smithi170.stdout: "threadpool_default_timeout": "60", 2024-03-27T09:34:47.012 INFO:teuthology.orchestra.run.smithi170.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-27T09:34:47.012 INFO:teuthology.orchestra.run.smithi170.stdout: "throttler_perf_counter": "true" 2024-03-27T09:34:47.012 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-27T09:34:47.014 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-27T09:34:47.122 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.120+0000 7f85427ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.126 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.125+0000 7f854159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.140 INFO:teuthology.orchestra.run.smithi170.stdout:{} 2024-03-27T09:34:47.141 INFO:teuthology.orchestra.run.smithi170.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-27T09:34:47.151 DEBUG:teuthology.orchestra.run.smithi170:> type ceph-objectstore-tool 2024-03-27T09:34:47.164 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T09:34:47.165 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-03-27T09:34:47.182 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-27T09:34:47.183 INFO:teuthology.run_tasks:Running task rados... 2024-03-27T09:34:47.194 INFO:tasks.rados:Beginning rados... 2024-03-27T09:34:47.194 INFO:tasks.rados:config is {'clients': ['client.0'], 'localize_reads': True, 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-03-27T09:34:47.194 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-sjust-crimson-testing-2024-03-20'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'ad2494b32de08d5bf12408a9a641b9ebfb123d09'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-matanb-crimson-thrash-osd', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6db07181a4df6107bd2ae48adf3d2421c02fa2c2'}} 2024-03-27T09:34:47.194 INFO:tasks.rados:config is {'clients': ['client.0'], 'localize_reads': True, 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-03-27T09:34:47.195 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-03-27T09:34:47.237 INFO:tasks.rados:joining rados 2024-03-27T09:34:47.237 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-27T09:34:47.237 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:34:47.238 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:34:47.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:34:47.238 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-27T09:34:47.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:34:47.239 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-03-27T09:34:47.239 INFO:tasks.rados:starting run 0 out of 1 2024-03-27T09:34:47.239 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-27T09:34:47.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-03-27T09:34:47.353 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.351+0000 7f63581d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.356 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.351+0000 7f2e611bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.358 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.357+0000 7f63581d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.360 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.351+0000 7f0ed3f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.360 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.360+0000 7f2e611bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.365 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.364+0000 7f0ed3f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.557 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:34:47.558 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:34:47.570 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-27T09:34:45.711212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107920,"kb_used_data":588,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379440,"statfs":{"total":386547056640,"available":386436546560,"internally_reserved":0,"allocated":602112,"data_stored":1873234,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960157"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.751972+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:34:43.751972+0000","last_peered":"2024-03-27T09:34:43.751972+0000","last_clean":"2024-03-27T09:34:43.751972+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:34:43.751972+0000","last_undegraded":"2024-03-27T09:34:43.751972+0000","last_fullsized":"2024-03-27T09:34:43.751972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.846268+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:34:42.846268+0000","last_peered":"2024-03-27T09:34:42.846268+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:34:42.846268+0000","last_undegraded":"2024-03-27T09:34:42.846268+0000","last_fullsized":"2024-03-27T09:34:42.846268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.824140+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:34:42.824140+0000","last_peered":"2024-03-27T09:34:42.824140+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:34:42.824140+0000","last_undegraded":"2024-03-27T09:34:42.824140+0000","last_fullsized":"2024-03-27T09:34:42.824140+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.822926+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:34:42.822926+0000","last_peered":"2024-03-27T09:34:42.822926+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:34:42.822926+0000","last_undegraded":"2024-03-27T09:34:42.822926+0000","last_fullsized":"2024-03-27T09:34:42.822926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826211+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:34:42.826211+0000","last_peered":"2024-03-27T09:34:42.826211+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:34:42.826211+0000","last_undegraded":"2024-03-27T09:34:42.826211+0000","last_fullsized":"2024-03-27T09:34:42.826211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.747843+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:34:43.747843+0000","last_peered":"2024-03-27T09:34:43.747843+0000","last_clean":"2024-03-27T09:34:43.747843+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:34:43.747843+0000","last_undegraded":"2024-03-27T09:34:43.747843+0000","last_fullsized":"2024-03-27T09:34:43.747843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:42.826142+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:34:42.826142+0000","last_peered":"2024-03-27T09:34:42.826142+0000","last_clean":"2024-03-27T09:34:41.724047+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:34:42.826142+0000","last_undegraded":"2024-03-27T09:34:42.826142+0000","last_fullsized":"2024-03-27T09:34:42.826142+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.752926+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:34:43.752926+0000","last_peered":"2024-03-27T09:34:43.752926+0000","last_clean":"2024-03-27T09:34:43.752926+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:34:43.752926+0000","last_undegraded":"2024-03-27T09:34:43.752926+0000","last_fullsized":"2024-03-27T09:34:43.752926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'96","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T09:34:43.395934+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:34:43.395934+0000","last_peered":"2024-03-27T09:34:43.395934+0000","last_clean":"2024-03-27T09:34:43.395934+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:34:43.395934+0000","last_undegraded":"2024-03-27T09:34:43.395934+0000","last_fullsized":"2024-03-27T09:34:43.395934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":615098,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:34:47.813 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'unique_pool_0' created 2024-03-27T09:34:47.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-27T09:34:47.970 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.959+0000 7f2642664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:47.974 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:47.973+0000 7f2642664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:48.816 INFO:teuthology.orchestra.run.smithi170.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-27T09:34:49.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:34:49.578 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:49.576+0000 7fb37faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:49.581 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:49.581+0000 7fb37faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:49.838 DEBUG:teuthology.orchestra.run.smithi170:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2024-03-27T09:34:49.871 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight read -> 100 2024-03-27T09:34:49.871 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight write -> 50 2024-03-27T09:34:49.871 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight delete -> 50 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight snap_create -> 50 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight snap_remove -> 50 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight write_excl -> 50 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout:ceph version 19.0.0-2247-gad2494b3 (ad2494b32de08d5bf12408a9a641b9ebfb123d09) squid (dev) 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout:Configuration: 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Number of operations: 4000 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Number of objects: 50 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Max in flight operations: 16 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Object size (in bytes): 4000000 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Write stride min: 400000 2024-03-27T09:34:49.872 INFO:tasks.rados.rados.0.smithi170.stdout: Write stride max: 800000 2024-03-27T09:34:50.820 INFO:tasks.rados.rados.0.smithi170.stdout:1: write initial oid 1 2024-03-27T09:34:50.821 INFO:tasks.rados.rados.0.smithi170.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-03-27T09:34:50.825 INFO:tasks.rados.rados.0.smithi170.stdout:1: writing smithi17031214-1 from 416807 to 892056 tid 1 2024-03-27T09:34:50.832 INFO:tasks.rados.rados.0.smithi170.stdout:1: writing smithi17031214-1 from 1542129 to 2016807 tid 2 2024-03-27T09:34:50.832 INFO:tasks.rados.rados.0.smithi170.stdout:2: write initial oid 2 2024-03-27T09:34:50.832 INFO:tasks.rados.rados.0.smithi170.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-03-27T09:34:50.838 INFO:tasks.rados.rados.0.smithi170.stdout:2: writing smithi17031214-2 from 416807 to 892056 tid 1 2024-03-27T09:34:50.845 INFO:tasks.rados.rados.0.smithi170.stdout:2: writing smithi17031214-2 from 1542129 to 2016807 tid 2 2024-03-27T09:34:50.845 INFO:tasks.rados.rados.0.smithi170.stdout:3: write initial oid 3 2024-03-27T09:34:50.845 INFO:tasks.rados.rados.0.smithi170.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-03-27T09:34:50.851 INFO:tasks.rados.rados.0.smithi170.stdout:3: writing smithi17031214-3 from 433614 to 984112 tid 1 2024-03-27T09:34:50.857 INFO:tasks.rados.rados.0.smithi170.stdout:3: writing smithi17031214-3 from 1600611 to 2033614 tid 2 2024-03-27T09:34:50.857 INFO:tasks.rados.rados.0.smithi170.stdout:4: write initial oid 4 2024-03-27T09:34:50.857 INFO:tasks.rados.rados.0.smithi170.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-03-27T09:34:50.863 INFO:tasks.rados.rados.0.smithi170.stdout:4: writing smithi17031214-4 from 450421 to 1076168 tid 1 2024-03-27T09:34:50.871 INFO:tasks.rados.rados.0.smithi170.stdout:4: writing smithi17031214-4 from 1659093 to 2050421 tid 2 2024-03-27T09:34:50.871 INFO:tasks.rados.rados.0.smithi170.stdout:5: write initial oid 5 2024-03-27T09:34:50.871 INFO:tasks.rados.rados.0.smithi170.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-03-27T09:34:50.880 INFO:tasks.rados.rados.0.smithi170.stdout:5: writing smithi17031214-5 from 467228 to 1168224 tid 1 2024-03-27T09:34:50.886 INFO:tasks.rados.rados.0.smithi170.stdout:5: writing smithi17031214-5 from 1717575 to 2067228 tid 2 2024-03-27T09:34:50.886 INFO:tasks.rados.rados.0.smithi170.stdout:6: write initial oid 6 2024-03-27T09:34:50.886 INFO:tasks.rados.rados.0.smithi170.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-03-27T09:34:50.893 INFO:tasks.rados.rados.0.smithi170.stdout:6: writing smithi17031214-6 from 484035 to 1260280 tid 1 2024-03-27T09:34:50.898 INFO:tasks.rados.rados.0.smithi170.stdout:6: writing smithi17031214-6 from 2059704 to 2084035 tid 2 2024-03-27T09:34:50.898 INFO:tasks.rados.rados.0.smithi170.stdout:7: write initial oid 7 2024-03-27T09:34:50.899 INFO:tasks.rados.rados.0.smithi170.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-03-27T09:34:50.906 INFO:tasks.rados.rados.0.smithi170.stdout:7: writing smithi17031214-7 from 500842 to 952336 tid 1 2024-03-27T09:34:50.914 INFO:tasks.rados.rados.0.smithi170.stdout:7: writing smithi17031214-7 from 1718186 to 2100842 tid 2 2024-03-27T09:34:50.914 INFO:tasks.rados.rados.0.smithi170.stdout:8: write initial oid 8 2024-03-27T09:34:50.915 INFO:tasks.rados.rados.0.smithi170.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-03-27T09:34:50.922 INFO:tasks.rados.rados.0.smithi170.stdout:8: writing smithi17031214-8 from 517649 to 1044392 tid 1 2024-03-27T09:34:50.930 INFO:tasks.rados.rados.0.smithi170.stdout:8: writing smithi17031214-8 from 1776668 to 2117649 tid 2 2024-03-27T09:34:50.930 INFO:tasks.rados.rados.0.smithi170.stdout:9: write initial oid 9 2024-03-27T09:34:50.930 INFO:tasks.rados.rados.0.smithi170.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-03-27T09:34:50.940 INFO:tasks.rados.rados.0.smithi170.stdout:9: writing smithi17031214-9 from 534456 to 1252801 tid 1 2024-03-27T09:34:50.946 INFO:tasks.rados.rados.0.smithi170.stdout:9: writing smithi17031214-9 from 1951503 to 2134456 tid 2 2024-03-27T09:34:50.947 INFO:tasks.rados.rados.0.smithi170.stdout:10: write initial oid 10 2024-03-27T09:34:50.947 INFO:tasks.rados.rados.0.smithi170.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-03-27T09:34:50.957 INFO:tasks.rados.rados.0.smithi170.stdout:10: writing smithi17031214-10 from 551263 to 1344857 tid 1 2024-03-27T09:34:50.963 INFO:tasks.rados.rados.0.smithi170.stdout:10: writing smithi17031214-10 from 1893632 to 2151263 tid 2 2024-03-27T09:34:50.963 INFO:tasks.rados.rados.0.smithi170.stdout:11: write initial oid 11 2024-03-27T09:34:50.963 INFO:tasks.rados.rados.0.smithi170.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-03-27T09:34:50.971 INFO:tasks.rados.rados.0.smithi170.stdout:11: writing smithi17031214-11 from 568070 to 1036913 tid 1 2024-03-27T09:34:50.980 INFO:tasks.rados.rados.0.smithi170.stdout:11: writing smithi17031214-11 from 1552114 to 2168070 tid 2 2024-03-27T09:34:50.980 INFO:tasks.rados.rados.0.smithi170.stdout:12: write initial oid 12 2024-03-27T09:34:50.980 INFO:tasks.rados.rados.0.smithi170.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-03-27T09:34:50.989 INFO:tasks.rados.rados.0.smithi170.stdout:12: writing smithi17031214-12 from 584877 to 1128969 tid 1 2024-03-27T09:34:50.997 INFO:tasks.rados.rados.0.smithi170.stdout:12: writing smithi17031214-12 from 1610596 to 2172599 tid 2 2024-03-27T09:34:50.997 INFO:tasks.rados.rados.0.smithi170.stdout:13: write initial oid 13 2024-03-27T09:34:50.998 INFO:tasks.rados.rados.0.smithi170.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-03-27T09:34:51.007 INFO:tasks.rados.rados.0.smithi170.stdout:13: writing smithi17031214-13 from 601684 to 1221025 tid 1 2024-03-27T09:34:51.016 INFO:tasks.rados.rados.0.smithi170.stdout:13: writing smithi17031214-13 from 1669078 to 2201684 tid 2 2024-03-27T09:34:51.016 INFO:tasks.rados.rados.0.smithi170.stdout:14: write initial oid 14 2024-03-27T09:34:51.016 INFO:tasks.rados.rados.0.smithi170.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-03-27T09:34:51.025 INFO:tasks.rados.rados.0.smithi170.stdout:14: writing smithi17031214-14 from 618491 to 1313081 tid 1 2024-03-27T09:34:51.031 INFO:tasks.rados.rados.0.smithi170.stdout:14: writing smithi17031214-14 from 2011207 to 2218491 tid 2 2024-03-27T09:34:51.031 INFO:tasks.rados.rados.0.smithi170.stdout:15: write initial oid 15 2024-03-27T09:34:51.031 INFO:tasks.rados.rados.0.smithi170.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-03-27T09:34:51.040 INFO:tasks.rados.rados.0.smithi170.stdout:15: writing smithi17031214-15 from 635298 to 1405137 tid 1 2024-03-27T09:34:51.046 INFO:tasks.rados.rados.0.smithi170.stdout:15: writing smithi17031214-15 from 2069689 to 2235298 tid 2 2024-03-27T09:34:51.046 INFO:tasks.rados.rados.0.smithi170.stdout:16: write initial oid 16 2024-03-27T09:34:51.046 INFO:tasks.rados.rados.0.smithi170.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-03-27T09:34:51.052 INFO:tasks.rados.rados.0.smithi170.stdout:16: writing smithi17031214-16 from 652105 to 1097193 tid 1 2024-03-27T09:34:51.060 INFO:tasks.rados.rados.0.smithi170.stdout:16: writing smithi17031214-16 from 1728171 to 2181159 tid 2 2024-03-27T09:34:51.060 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.060 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 1 to smithi17031214-3 2024-03-27T09:34:51.060 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 1 to smithi17031214-4 2024-03-27T09:34:51.060 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 1 to smithi17031214-1 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 1 to smithi17031214-7 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 1 to smithi17031214-5 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 2 to smithi17031214-3 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 1 to smithi17031214-2 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 2 to smithi17031214-1 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 2 to smithi17031214-5 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 3 to smithi17031214-5 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 1 to smithi17031214-9 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 4 to smithi17031214-5 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: oid 5 updating version 0 to 3 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: oid 5 version 3 is already newer than 3 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:5: oid 5 version 3 is already newer than 1 2024-03-27T09:34:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 3 to smithi17031214-3 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 4 to smithi17031214-3 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: oid 3 updating version 0 to 3 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: oid 3 version 3 is already newer than 1 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: oid 3 version 3 is already newer than 2 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:3: done (15 left) 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:5: done (14 left) 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:17: write initial oid 17 2024-03-27T09:34:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-03-27T09:34:51.067 INFO:tasks.rados.rados.0.smithi170.stdout:17: writing smithi17031214-17 from 668912 to 1305602 tid 1 2024-03-27T09:34:51.072 INFO:tasks.rados.rados.0.smithi170.stdout:17: writing smithi17031214-17 from 1903006 to 2268912 tid 2 2024-03-27T09:34:51.072 INFO:tasks.rados.rados.0.smithi170.stdout:18: write initial oid 18 2024-03-27T09:34:51.072 INFO:tasks.rados.rados.0.smithi170.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-03-27T09:34:51.080 INFO:tasks.rados.rados.0.smithi170.stdout:18: writing smithi17031214-18 from 685719 to 1397658 tid 1 2024-03-27T09:34:51.088 INFO:tasks.rados.rados.0.smithi170.stdout:18: writing smithi17031214-18 from 1845135 to 2285719 tid 2 2024-03-27T09:34:51.088 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.088 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 2 to smithi17031214-7 2024-03-27T09:34:51.088 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 1 to smithi17031214-11 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 2 to smithi17031214-4 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 1 to smithi17031214-10 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 3 to smithi17031214-1 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 1 to smithi17031214-13 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 4 to smithi17031214-1 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: oid 1 updating version 0 to 2 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: oid 1 version 2 is already newer than 1 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: oid 1 updating version 2 to 3 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 2 to smithi17031214-2 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 2 to smithi17031214-9 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:1: done (15 left) 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:19: write initial oid 19 2024-03-27T09:34:51.089 INFO:tasks.rados.rados.0.smithi170.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-03-27T09:34:51.098 INFO:tasks.rados.rados.0.smithi170.stdout:19: writing smithi17031214-19 from 702526 to 1489714 tid 1 2024-03-27T09:34:51.103 INFO:tasks.rados.rados.0.smithi170.stdout:19: writing smithi17031214-19 from 1903617 to 2302526 tid 2 2024-03-27T09:34:51.103 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.103 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 3 to smithi17031214-2 2024-03-27T09:34:51.103 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 1 to smithi17031214-15 2024-03-27T09:34:51.104 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 3 to smithi17031214-9 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 4 to smithi17031214-9 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: oid 9 updating version 0 to 3 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: oid 9 version 3 is already newer than 1 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: oid 9 version 3 is already newer than 3 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:9: done (15 left) 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:20: write initial oid 20 2024-03-27T09:34:51.106 INFO:tasks.rados.rados.0.smithi170.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-03-27T09:34:51.112 INFO:tasks.rados.rados.0.smithi170.stdout:20: writing smithi17031214-20 from 719333 to 1181770 tid 1 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:20: writing smithi17031214-20 from 1962099 to 2319333 tid 2 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 4 to smithi17031214-2 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:2: oid 2 updating version 0 to 3 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:2: oid 2 version 3 is already newer than 1 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:2: oid 2 version 3 is already newer than 3 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-27T09:34:51.119 INFO:tasks.rados.rados.0.smithi170.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-03-27T09:34:51.120 INFO:tasks.rados.rados.0.smithi170.stdout:2: done (15 left) 2024-03-27T09:34:51.120 INFO:tasks.rados.rados.0.smithi170.stdout:21: write initial oid 21 2024-03-27T09:34:51.120 INFO:tasks.rados.rados.0.smithi170.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-03-27T09:34:51.128 INFO:tasks.rados.rados.0.smithi170.stdout:21: writing smithi17031214-21 from 736140 to 1273826 tid 1 2024-03-27T09:34:51.136 INFO:tasks.rados.rados.0.smithi170.stdout:21: writing smithi17031214-21 from 2020581 to 2336140 tid 2 2024-03-27T09:34:51.136 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.136 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 1 to smithi17031214-8 2024-03-27T09:34:51.136 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 1 to smithi17031214-18 2024-03-27T09:34:51.136 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 2 to smithi17031214-13 2024-03-27T09:34:51.137 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 2 to smithi17031214-10 2024-03-27T09:34:51.138 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 3 to smithi17031214-7 2024-03-27T09:34:51.141 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 1 to smithi17031214-6 2024-03-27T09:34:51.143 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 2 to smithi17031214-11 2024-03-27T09:34:51.152 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 3 to smithi17031214-4 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 4 to smithi17031214-7 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: oid 7 updating version 0 to 1 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: oid 7 updating version 1 to 3 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: oid 7 version 3 is already newer than 2 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 4 to smithi17031214-4 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: oid 4 updating version 0 to 1 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: oid 4 updating version 1 to 2 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: oid 4 updating version 2 to 3 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:4: done (15 left) 2024-03-27T09:34:51.153 INFO:tasks.rados.rados.0.smithi170.stdout:7: done (14 left) 2024-03-27T09:34:51.154 INFO:tasks.rados.rados.0.smithi170.stdout:22: write initial oid 22 2024-03-27T09:34:51.154 INFO:tasks.rados.rados.0.smithi170.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-03-27T09:34:51.161 INFO:tasks.rados.rados.0.smithi170.stdout:22: writing smithi17031214-22 from 752947 to 1365882 tid 1 2024-03-27T09:34:51.167 INFO:tasks.rados.rados.0.smithi170.stdout:22: writing smithi17031214-22 from 1962710 to 2352947 tid 2 2024-03-27T09:34:51.167 INFO:tasks.rados.rados.0.smithi170.stdout:23: write initial oid 23 2024-03-27T09:34:51.167 INFO:tasks.rados.rados.0.smithi170.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-03-27T09:34:51.175 INFO:tasks.rados.rados.0.smithi170.stdout:23: writing smithi17031214-23 from 769754 to 1457938 tid 1 2024-03-27T09:34:51.182 INFO:tasks.rados.rados.0.smithi170.stdout:23: writing smithi17031214-23 from 2021192 to 2369754 tid 2 2024-03-27T09:34:51.182 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.183 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 2 to smithi17031214-15 2024-03-27T09:34:51.183 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 2 to smithi17031214-6 2024-03-27T09:34:51.183 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 2 to smithi17031214-18 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 3 to smithi17031214-18 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 4 to smithi17031214-18 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: oid 18 updating version 0 to 1 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: oid 18 updating version 1 to 3 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: oid 18 version 3 is already newer than 3 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-27T09:34:51.186 INFO:tasks.rados.rados.0.smithi170.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-03-27T09:34:51.187 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 3 to smithi17031214-10 2024-03-27T09:34:51.187 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 2 to smithi17031214-8 2024-03-27T09:34:51.187 INFO:tasks.rados.rados.0.smithi170.stdout:18: done (15 left) 2024-03-27T09:34:51.187 INFO:tasks.rados.rados.0.smithi170.stdout:24: write initial oid 24 2024-03-27T09:34:51.187 INFO:tasks.rados.rados.0.smithi170.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-03-27T09:34:51.192 INFO:tasks.rados.rados.0.smithi170.stdout:24: writing smithi17031214-24 from 786561 to 1266347 tid 1 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:24: writing smithi17031214-24 from 1796027 to 2263691 tid 2 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 4 to smithi17031214-10 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: oid 10 updating version 0 to 3 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: oid 10 version 3 is already newer than 1 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: oid 10 version 3 is already newer than 2 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:10: done (15 left) 2024-03-27T09:34:51.199 INFO:tasks.rados.rados.0.smithi170.stdout:25: write initial oid 25 2024-03-27T09:34:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-03-27T09:34:51.205 INFO:tasks.rados.rados.0.smithi170.stdout:25: writing smithi17031214-25 from 403368 to 958403 tid 1 2024-03-27T09:34:51.213 INFO:tasks.rados.rados.0.smithi170.stdout:25: writing smithi17031214-25 from 1454509 to 2182184 tid 2 2024-03-27T09:34:51.213 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.213 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 1 to smithi17031214-19 2024-03-27T09:34:51.213 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 3 to smithi17031214-6 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 4 to smithi17031214-6 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: oid 6 updating version 0 to 4 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: oid 6 updating version 4 to 6 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: oid 6 version 6 is already newer than 6 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 3 to smithi17031214-13 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 1 to smithi17031214-23 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 3 to smithi17031214-15 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 1 to smithi17031214-16 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 3 to smithi17031214-11 2024-03-27T09:34:51.214 INFO:tasks.rados.rados.0.smithi170.stdout:6: done (15 left) 2024-03-27T09:34:51.215 INFO:tasks.rados.rados.0.smithi170.stdout:26: write initial oid 26 2024-03-27T09:34:51.215 INFO:tasks.rados.rados.0.smithi170.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-03-27T09:34:51.218 INFO:tasks.rados.rados.0.smithi170.stdout:26: writing smithi17031214-26 from 420175 to 1050459 tid 1 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:26: writing smithi17031214-26 from 1796638 to 2267971 tid 2 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 4 to smithi17031214-13 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:13: oid 13 updating version 0 to 2 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:13: oid 13 version 2 is already newer than 1 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:13: oid 13 updating version 2 to 3 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 4 to smithi17031214-15 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:15: oid 15 updating version 0 to 2 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:15: oid 15 updating version 2 to 3 2024-03-27T09:34:51.228 INFO:tasks.rados.rados.0.smithi170.stdout:15: oid 15 version 3 is already newer than 1 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 4 to smithi17031214-11 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: oid 11 updating version 0 to 1 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: oid 11 updating version 1 to 2 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: oid 11 updating version 2 to 3 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:11: done (15 left) 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:13: done (14 left) 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:15: done (13 left) 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:27: write initial oid 27 2024-03-27T09:34:51.229 INFO:tasks.rados.rados.0.smithi170.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-03-27T09:34:51.235 INFO:tasks.rados.rados.0.smithi170.stdout:27: writing smithi17031214-27 from 436982 to 1142515 tid 1 2024-03-27T09:34:51.243 INFO:tasks.rados.rados.0.smithi170.stdout:27: writing smithi17031214-27 from 1855120 to 2436982 tid 2 2024-03-27T09:34:51.243 INFO:tasks.rados.rados.0.smithi170.stdout:28: write initial oid 28 2024-03-27T09:34:51.243 INFO:tasks.rados.rados.0.smithi170.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-03-27T09:34:51.250 INFO:tasks.rados.rados.0.smithi170.stdout:28: writing smithi17031214-28 from 453789 to 1234571 tid 1 2024-03-27T09:34:51.259 INFO:tasks.rados.rados.0.smithi170.stdout:28: writing smithi17031214-28 from 1913602 to 2388604 tid 2 2024-03-27T09:34:51.259 INFO:tasks.rados.rados.0.smithi170.stdout:29: write initial oid 29 2024-03-27T09:34:51.259 INFO:tasks.rados.rados.0.smithi170.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-03-27T09:34:51.265 INFO:tasks.rados.rados.0.smithi170.stdout:29: writing smithi17031214-29 from 470596 to 926627 tid 1 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout:29: writing smithi17031214-29 from 1572084 to 2190744 tid 2 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 2 to smithi17031214-23 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 2 to smithi17031214-19 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 3 to smithi17031214-8 2024-03-27T09:34:51.274 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 1 to smithi17031214-22 2024-03-27T09:34:51.278 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 4 to smithi17031214-8 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:8: oid 8 updating version 0 to 2 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:8: oid 8 version 2 is already newer than 1 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:8: oid 8 updating version 2 to 3 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:8: done (15 left) 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:30: write initial oid 30 2024-03-27T09:34:51.279 INFO:tasks.rados.rados.0.smithi170.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-03-27T09:34:51.285 INFO:tasks.rados.rados.0.smithi170.stdout:30: writing smithi17031214-30 from 487403 to 1018683 tid 1 2024-03-27T09:34:51.290 INFO:tasks.rados.rados.0.smithi170.stdout:30: writing smithi17031214-30 from 1514213 to 1992884 tid 2 2024-03-27T09:34:51.291 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.291 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 1 to smithi17031214-24 2024-03-27T09:34:51.291 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 1 to smithi17031214-14 2024-03-27T09:34:51.294 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 3 to smithi17031214-23 2024-03-27T09:34:51.296 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 2 to smithi17031214-16 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 4 to smithi17031214-23 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: oid 23 updating version 0 to 5 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: oid 23 version 5 is already newer than 4 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: oid 23 updating version 5 to 6 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:23: done (15 left) 2024-03-27T09:34:51.300 INFO:tasks.rados.rados.0.smithi170.stdout:31: write initial oid 31 2024-03-27T09:34:51.301 INFO:tasks.rados.rados.0.smithi170.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-03-27T09:34:51.307 INFO:tasks.rados.rados.0.smithi170.stdout:31: writing smithi17031214-31 from 504210 to 1110739 tid 1 2024-03-27T09:34:51.314 INFO:tasks.rados.rados.0.smithi170.stdout:31: writing smithi17031214-31 from 1572695 to 2195024 tid 2 2024-03-27T09:34:51.314 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.315 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 1 to smithi17031214-28 2024-03-27T09:34:51.316 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 2 to smithi17031214-24 2024-03-27T09:34:51.319 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 1 to smithi17031214-12 2024-03-27T09:34:51.323 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 1 to smithi17031214-25 2024-03-27T09:34:51.338 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 3 to smithi17031214-24 2024-03-27T09:34:51.341 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 4 to smithi17031214-24 2024-03-27T09:34:51.341 INFO:tasks.rados.rados.0.smithi170.stdout:24: oid 24 updating version 0 to 6 2024-03-27T09:34:51.341 INFO:tasks.rados.rados.0.smithi170.stdout:24: oid 24 version 6 is already newer than 5 2024-03-27T09:34:51.341 INFO:tasks.rados.rados.0.smithi170.stdout:24: oid 24 version 6 is already newer than 4 2024-03-27T09:34:51.341 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-27T09:34:51.342 INFO:tasks.rados.rados.0.smithi170.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-03-27T09:34:51.342 INFO:tasks.rados.rados.0.smithi170.stdout:24: done (15 left) 2024-03-27T09:34:51.342 INFO:tasks.rados.rados.0.smithi170.stdout:32: write initial oid 32 2024-03-27T09:34:51.342 INFO:tasks.rados.rados.0.smithi170.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-03-27T09:34:51.348 INFO:tasks.rados.rados.0.smithi170.stdout:32: writing smithi17031214-32 from 521017 to 1319148 tid 1 2024-03-27T09:34:51.354 INFO:tasks.rados.rados.0.smithi170.stdout:32: writing smithi17031214-32 from 1747530 to 2229870 tid 2 2024-03-27T09:34:51.354 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.356 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 2 to smithi17031214-22 2024-03-27T09:34:51.359 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 2 to smithi17031214-14 2024-03-27T09:34:51.360 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 1 to smithi17031214-17 2024-03-27T09:34:51.362 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 3 to smithi17031214-19 2024-03-27T09:34:51.368 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 3 to smithi17031214-22 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 4 to smithi17031214-22 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: oid 22 updating version 0 to 9 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: oid 22 version 9 is already newer than 7 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: oid 22 version 9 is already newer than 9 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:22: done (15 left) 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:33: write initial oid 33 2024-03-27T09:34:51.370 INFO:tasks.rados.rados.0.smithi170.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-03-27T09:34:51.377 INFO:tasks.rados.rados.0.smithi170.stdout:33: writing smithi17031214-33 from 537824 to 1011204 tid 1 2024-03-27T09:34:51.386 INFO:tasks.rados.rados.0.smithi170.stdout:33: writing smithi17031214-33 from 1806012 to 2432010 tid 2 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 3 to smithi17031214-14 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 4 to smithi17031214-14 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: oid 14 updating version 0 to 6 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: oid 14 version 6 is already newer than 4 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: oid 14 version 6 is already newer than 6 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 4 to smithi17031214-19 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:19: oid 19 updating version 0 to 2 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:19: oid 19 version 2 is already newer than 1 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:19: oid 19 updating version 2 to 3 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-27T09:34:51.387 INFO:tasks.rados.rados.0.smithi170.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-03-27T09:34:51.388 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 2 to smithi17031214-12 2024-03-27T09:34:51.388 INFO:tasks.rados.rados.0.smithi170.stdout:14: done (15 left) 2024-03-27T09:34:51.388 INFO:tasks.rados.rados.0.smithi170.stdout:19: done (14 left) 2024-03-27T09:34:51.388 INFO:tasks.rados.rados.0.smithi170.stdout:34: write initial oid 34 2024-03-27T09:34:51.388 INFO:tasks.rados.rados.0.smithi170.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-03-27T09:34:51.392 INFO:tasks.rados.rados.0.smithi170.stdout:34: writing smithi17031214-34 from 554631 to 1103260 tid 1 2024-03-27T09:34:51.401 INFO:tasks.rados.rados.0.smithi170.stdout:34: writing smithi17031214-34 from 1748141 to 2234150 tid 2 2024-03-27T09:34:51.401 INFO:tasks.rados.rados.0.smithi170.stdout:35: write initial oid 35 2024-03-27T09:34:51.401 INFO:tasks.rados.rados.0.smithi170.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-03-27T09:34:51.411 INFO:tasks.rados.rados.0.smithi170.stdout:35: writing smithi17031214-35 from 571438 to 1195316 tid 1 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:35: writing smithi17031214-35 from 1806623 to 2436290 tid 2 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 3 to smithi17031214-12 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 3 to smithi17031214-16 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 4 to smithi17031214-16 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: oid 16 updating version 0 to 6 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: oid 16 version 6 is already newer than 5 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: oid 16 version 6 is already newer than 4 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:34:51.423 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 2 to smithi17031214-28 2024-03-27T09:34:51.424 INFO:tasks.rados.rados.0.smithi170.stdout:16: done (15 left) 2024-03-27T09:34:51.424 INFO:tasks.rados.rados.0.smithi170.stdout:36: write initial oid 36 2024-03-27T09:34:51.424 INFO:tasks.rados.rados.0.smithi170.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-03-27T09:34:51.432 INFO:tasks.rados.rados.0.smithi170.stdout:36: writing smithi17031214-36 from 588245 to 1287372 tid 1 2024-03-27T09:34:51.440 INFO:tasks.rados.rados.0.smithi170.stdout:36: writing smithi17031214-36 from 1865105 to 2354783 tid 2 2024-03-27T09:34:51.440 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.440 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 4 to smithi17031214-12 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:12: oid 12 updating version 0 to 6 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:12: oid 12 version 6 is already newer than 6 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:12: oid 12 version 6 is already newer than 4 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-03-27T09:34:51.441 INFO:tasks.rados.rados.0.smithi170.stdout:12: done (15 left) 2024-03-27T09:34:51.442 INFO:tasks.rados.rados.0.smithi170.stdout:37: write initial oid 37 2024-03-27T09:34:51.442 INFO:tasks.rados.rados.0.smithi170.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-03-27T09:34:51.450 INFO:tasks.rados.rados.0.smithi170.stdout:37: writing smithi17031214-37 from 605052 to 1379428 tid 1 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:37: writing smithi17031214-37 from 1923587 to 2556923 tid 2 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 2 to smithi17031214-25 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 1 to smithi17031214-30 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 1 to smithi17031214-34 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 1 to smithi17031214-21 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 2 to smithi17031214-17 2024-03-27T09:34:51.459 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 3 to smithi17031214-25 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 4 to smithi17031214-25 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: oid 25 updating version 0 to 4 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: oid 25 updating version 4 to 5 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: oid 25 updating version 5 to 6 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:25: done (15 left) 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:38: write initial oid 38 2024-03-27T09:34:51.460 INFO:tasks.rados.rados.0.smithi170.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-03-27T09:34:51.466 INFO:tasks.rados.rados.0.smithi170.stdout:38: writing smithi17031214-38 from 621859 to 1071484 tid 1 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:38: writing smithi17031214-38 from 1865716 to 2621859 tid 2 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 1 to smithi17031214-20 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 1 to smithi17031214-33 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 3 to smithi17031214-28 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 2 to smithi17031214-34 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 4 to smithi17031214-28 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:28: oid 28 updating version 0 to 6 2024-03-27T09:34:51.476 INFO:tasks.rados.rados.0.smithi170.stdout:28: oid 28 version 6 is already newer than 4 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:28: oid 28 version 6 is already newer than 5 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:28: done (15 left) 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:39: write initial oid 39 2024-03-27T09:34:51.477 INFO:tasks.rados.rados.0.smithi170.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-03-27T09:34:51.482 INFO:tasks.rados.rados.0.smithi170.stdout:39: writing smithi17031214-39 from 638666 to 1163540 tid 1 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout:39: writing smithi17031214-39 from 1924198 to 2561203 tid 2 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 3 to smithi17031214-34 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 4 to smithi17031214-34 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout:34: oid 34 updating version 0 to 9 2024-03-27T09:34:51.491 INFO:tasks.rados.rados.0.smithi170.stdout:34: oid 34 version 9 is already newer than 9 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:34: oid 34 version 9 is already newer than 7 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:34: done (15 left) 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:40: write initial oid 40 2024-03-27T09:34:51.492 INFO:tasks.rados.rados.0.smithi170.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-03-27T09:34:51.499 INFO:tasks.rados.rados.0.smithi170.stdout:40: writing smithi17031214-40 from 655473 to 1371949 tid 1 2024-03-27T09:34:51.506 INFO:tasks.rados.rados.0.smithi170.stdout:40: writing smithi17031214-40 from 2099033 to 2655473 tid 2 2024-03-27T09:34:51.506 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.508 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 3 to smithi17031214-17 2024-03-27T09:34:51.509 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 2 to smithi17031214-30 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 4 to smithi17031214-17 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: oid 17 updating version 0 to 4 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: oid 17 updating version 4 to 6 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: oid 17 version 6 is already newer than 5 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:17: done (15 left) 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:41: write initial oid 41 2024-03-27T09:34:51.513 INFO:tasks.rados.rados.0.smithi170.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-03-27T09:34:51.522 INFO:tasks.rados.rados.0.smithi170.stdout:41: writing smithi17031214-41 from 672280 to 1464005 tid 1 2024-03-27T09:34:51.530 INFO:tasks.rados.rados.0.smithi170.stdout:41: writing smithi17031214-41 from 2157515 to 2672280 tid 2 2024-03-27T09:34:51.530 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.531 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 2 to smithi17031214-21 2024-03-27T09:34:51.531 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 1 to smithi17031214-27 2024-03-27T09:34:51.531 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 2 to smithi17031214-20 2024-03-27T09:34:51.531 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 3 to smithi17031214-30 2024-03-27T09:34:51.532 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 3 to smithi17031214-21 2024-03-27T09:34:51.535 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 2 to smithi17031214-33 2024-03-27T09:34:51.536 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 4 to smithi17031214-30 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:30: oid 30 updating version 0 to 9 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:30: oid 30 version 9 is already newer than 9 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:30: oid 30 version 9 is already newer than 7 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 1 to smithi17031214-39 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:30: done (15 left) 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:42: write initial oid 42 2024-03-27T09:34:51.537 INFO:tasks.rados.rados.0.smithi170.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-03-27T09:34:51.543 INFO:tasks.rados.rados.0.smithi170.stdout:42: writing smithi17031214-42 from 689087 to 1156061 tid 1 2024-03-27T09:34:51.553 INFO:tasks.rados.rados.0.smithi170.stdout:42: writing smithi17031214-42 from 1699644 to 2483976 tid 2 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 4 to smithi17031214-21 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: oid 21 updating version 0 to 4 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: oid 21 updating version 4 to 6 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: oid 21 version 6 is already newer than 6 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 3 to smithi17031214-20 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 1 to smithi17031214-32 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:21: done (15 left) 2024-03-27T09:34:51.554 INFO:tasks.rados.rados.0.smithi170.stdout:43: write initial oid 43 2024-03-27T09:34:51.555 INFO:tasks.rados.rados.0.smithi170.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-03-27T09:34:51.563 INFO:tasks.rados.rados.0.smithi170.stdout:43: writing smithi17031214-43 from 705894 to 1248117 tid 1 2024-03-27T09:34:51.573 INFO:tasks.rados.rados.0.smithi170.stdout:43: writing smithi17031214-43 from 1758126 to 2402469 tid 2 2024-03-27T09:34:51.573 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 4 to smithi17031214-20 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: oid 20 updating version 0 to 9 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: oid 20 version 9 is already newer than 7 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: oid 20 version 9 is already newer than 9 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 2 to smithi17031214-39 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:20: done (15 left) 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:44: write initial oid 44 2024-03-27T09:34:51.574 INFO:tasks.rados.rados.0.smithi170.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-03-27T09:34:51.582 INFO:tasks.rados.rados.0.smithi170.stdout:44: writing smithi17031214-44 from 722701 to 1340173 tid 1 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout:44: writing smithi17031214-44 from 1816608 to 2604609 tid 2 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 2 to smithi17031214-27 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 1 to smithi17031214-40 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 3 to smithi17031214-27 2024-03-27T09:34:51.591 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 1 to smithi17031214-26 2024-03-27T09:34:51.593 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 3 to smithi17031214-33 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 4 to smithi17031214-33 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: oid 33 updating version 0 to 6 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: oid 33 version 6 is already newer than 5 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: oid 33 version 6 is already newer than 4 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:33: done (15 left) 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:45: write initial oid 45 2024-03-27T09:34:51.595 INFO:tasks.rados.rados.0.smithi170.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-03-27T09:34:51.605 INFO:tasks.rados.rados.0.smithi170.stdout:45: writing smithi17031214-45 from 739508 to 1432229 tid 1 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:45: writing smithi17031214-45 from 1875090 to 2523102 tid 2 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 3 to smithi17031214-39 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 4 to smithi17031214-27 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:27: oid 27 updating version 0 to 9 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:27: oid 27 version 9 is already newer than 7 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:27: oid 27 version 9 is already newer than 9 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 1 to smithi17031214-36 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 4 to smithi17031214-39 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:39: oid 39 updating version 0 to 4 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:39: oid 39 updating version 4 to 6 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:39: oid 39 version 6 is already newer than 5 2024-03-27T09:34:51.616 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 1 to smithi17031214-29 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:27: done (15 left) 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:39: done (14 left) 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:46: write initial oid 46 2024-03-27T09:34:51.617 INFO:tasks.rados.rados.0.smithi170.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-03-27T09:34:51.627 INFO:tasks.rados.rados.0.smithi170.stdout:46: writing smithi17031214-46 from 756315 to 1524285 tid 1 2024-03-27T09:34:51.637 INFO:tasks.rados.rados.0.smithi170.stdout:46: writing smithi17031214-46 from 1933572 to 2725242 tid 2 2024-03-27T09:34:51.637 INFO:tasks.rados.rados.0.smithi170.stdout:47: write initial oid 47 2024-03-27T09:34:51.638 INFO:tasks.rados.rados.0.smithi170.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-03-27T09:34:51.647 INFO:tasks.rados.rados.0.smithi170.stdout:47: writing smithi17031214-47 from 773122 to 1332694 tid 1 2024-03-27T09:34:51.656 INFO:tasks.rados.rados.0.smithi170.stdout:47: writing smithi17031214-47 from 1992054 to 2643735 tid 2 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 2 to smithi17031214-40 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 2 to smithi17031214-32 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 3 to smithi17031214-32 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 4 to smithi17031214-32 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: oid 32 updating version 0 to 9 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: oid 32 version 9 is already newer than 9 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: oid 32 version 9 is already newer than 7 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 2 to smithi17031214-26 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:32: done (15 left) 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:48: write initial oid 48 2024-03-27T09:34:51.657 INFO:tasks.rados.rados.0.smithi170.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-03-27T09:34:51.666 INFO:tasks.rados.rados.0.smithi170.stdout:48: writing smithi17031214-48 from 789929 to 1424750 tid 1 2024-03-27T09:34:51.679 INFO:tasks.rados.rados.0.smithi170.stdout:48: writing smithi17031214-48 from 2050536 to 2789929 tid 2 2024-03-27T09:34:51.679 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.679 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 3 to smithi17031214-40 2024-03-27T09:34:51.679 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 4 to smithi17031214-40 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:40: oid 40 updating version 0 to 11 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:40: oid 40 updating version 11 to 12 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:40: oid 40 version 12 is already newer than 10 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 3 to smithi17031214-26 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 1 to smithi17031214-43 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 2 to smithi17031214-29 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:40: done (15 left) 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:49: write initial oid 49 2024-03-27T09:34:51.680 INFO:tasks.rados.rados.0.smithi170.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-03-27T09:34:51.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:34:51.687 INFO:tasks.rados.rados.0.smithi170.stdout:49: writing smithi17031214-49 from 406736 to 1116806 tid 1 2024-03-27T09:34:51.698 INFO:tasks.rados.rados.0.smithi170.stdout:49: writing smithi17031214-49 from 1709018 to 2364368 tid 2 2024-03-27T09:34:51.698 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.698 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 3 to smithi17031214-29 2024-03-27T09:34:51.698 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 1 to smithi17031214-47 2024-03-27T09:34:51.700 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 2 to smithi17031214-36 2024-03-27T09:34:51.702 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 1 to smithi17031214-42 2024-03-27T09:34:51.704 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 1 to smithi17031214-35 2024-03-27T09:34:51.705 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 4 to smithi17031214-26 2024-03-27T09:34:51.705 INFO:tasks.rados.rados.0.smithi170.stdout:26: oid 26 updating version 0 to 9 2024-03-27T09:34:51.705 INFO:tasks.rados.rados.0.smithi170.stdout:26: oid 26 version 9 is already newer than 7 2024-03-27T09:34:51.705 INFO:tasks.rados.rados.0.smithi170.stdout:26: oid 26 version 9 is already newer than 9 2024-03-27T09:34:51.705 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-27T09:34:51.706 INFO:tasks.rados.rados.0.smithi170.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-03-27T09:34:51.706 INFO:tasks.rados.rados.0.smithi170.stdout:26: done (15 left) 2024-03-27T09:34:51.706 INFO:tasks.rados.rados.0.smithi170.stdout:50: write initial oid 50 2024-03-27T09:34:51.706 INFO:tasks.rados.rados.0.smithi170.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-03-27T09:34:51.713 INFO:tasks.rados.rados.0.smithi170.stdout:50: writing smithi17031214-50 from 423543 to 1208862 tid 1 2024-03-27T09:34:51.725 INFO:tasks.rados.rados.0.smithi170.stdout:50: writing smithi17031214-50 from 1767500 to 2566508 tid 2 2024-03-27T09:34:51.725 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 2 to smithi17031214-47 2024-03-27T09:34:51.725 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 4 to smithi17031214-29 2024-03-27T09:34:51.725 INFO:tasks.rados.rados.0.smithi170.stdout:29: oid 29 updating version 0 to 10 2024-03-27T09:34:51.725 INFO:tasks.rados.rados.0.smithi170.stdout:29: oid 29 updating version 10 to 12 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:29: oid 29 version 12 is already newer than 12 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:29: done (15 left) 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:51: write (excl) oid 2 current snap is 0 2024-03-27T09:34:51.726 INFO:tasks.rados.rados.0.smithi170.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-03-27T09:34:51.731 INFO:tasks.rados.rados.0.smithi170.stdout:51: writing smithi17031214-2 from 440350 to 900918 tid 1 2024-03-27T09:34:51.737 INFO:tasks.rados.rados.0.smithi170.stdout:51: writing smithi17031214-2 from 1309629 to 1852295 tid 2 2024-03-27T09:34:51.743 INFO:tasks.rados.rados.0.smithi170.stdout:51: writing smithi17031214-2 from 2323973 to 2840350 tid 3 2024-03-27T09:34:51.743 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.743 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 2 to smithi17031214-43 2024-03-27T09:34:51.744 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 3 to smithi17031214-36 2024-03-27T09:34:51.748 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 3 to smithi17031214-43 2024-03-27T09:34:51.749 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 1 to smithi17031214-38 2024-03-27T09:34:51.749 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 4 to smithi17031214-36 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:36: oid 36 updating version 0 to 9 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:36: oid 36 version 9 is already newer than 7 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:36: oid 36 version 9 is already newer than 9 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:36: done (15 left) 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:52: write (excl) oid 14 current snap is 0 2024-03-27T09:34:51.750 INFO:tasks.rados.rados.0.smithi170.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-03-27T09:34:51.757 INFO:tasks.rados.rados.0.smithi170.stdout:52: writing smithi17031214-14 from 457157 to 992974 tid 1 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:52: writing smithi17031214-14 from 1768111 to 2170788 tid 2 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 1 to smithi17031214-48 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 2 to smithi17031214-42 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 3 to smithi17031214-42 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 3 to smithi17031214-47 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 4 to smithi17031214-47 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:47: oid 47 updating version 0 to 8 2024-03-27T09:34:51.766 INFO:tasks.rados.rados.0.smithi170.stdout:47: oid 47 updating version 8 to 9 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:47: oid 47 version 9 is already newer than 7 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:47: done (15 left) 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:53: read oid 20 snap -1 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-27T09:34:51.767 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 4 to smithi17031214-43 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: oid 43 updating version 0 to 10 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: oid 43 updating version 10 to 12 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: oid 43 version 12 is already newer than 12 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:43: done (15 left) 2024-03-27T09:34:51.770 INFO:tasks.rados.rados.0.smithi170.stdout:54: delete oid 23 current snap is 0 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 4 to smithi17031214-42 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:42: oid 42 updating version 0 to 15 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:42: oid 42 version 15 is already newer than 13 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:42: oid 42 version 15 is already newer than 15 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 15 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-27T09:34:51.779 INFO:tasks.rados.rados.0.smithi170.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-03-27T09:34:51.800 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 1 to smithi17031214-31 2024-03-27T09:34:51.810 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 2 to smithi17031214-35 2024-03-27T09:34:51.813 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 3 to smithi17031214-35 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 4 to smithi17031214-35 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:35: oid 35 updating version 0 to 12 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:35: oid 35 version 12 is already newer than 12 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:35: oid 35 version 12 is already newer than 10 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-27T09:34:51.817 INFO:tasks.rados.rados.0.smithi170.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-03-27T09:34:51.832 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 2 to smithi17031214-48 2024-03-27T09:34:51.832 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 1 to smithi17031214-46 2024-03-27T09:34:51.837 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 2 to smithi17031214-38 2024-03-27T09:34:51.838 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 1 to smithi17031214-44 2024-03-27T09:34:51.838 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 3 to smithi17031214-38 2024-03-27T09:34:51.841 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 1 to smithi17031214-45 2024-03-27T09:34:51.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:51.838+0000 7fdfb5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 4 to smithi17031214-38 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:38: oid 38 updating version 0 to 12 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:38: oid 38 version 12 is already newer than 10 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:38: oid 38 version 12 is already newer than 12 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-27T09:34:51.844 INFO:tasks.rados.rados.0.smithi170.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-03-27T09:34:51.847 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 3 to smithi17031214-48 2024-03-27T09:34:51.849 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 4 to smithi17031214-48 2024-03-27T09:34:51.849 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 updating version 0 to 9 2024-03-27T09:34:51.849 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 version 9 is already newer than 7 2024-03-27T09:34:51.849 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 version 9 is already newer than 9 2024-03-27T09:34:51.850 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-27T09:34:51.850 INFO:tasks.rados.rados.0.smithi170.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-03-27T09:34:51.850 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 2 to smithi17031214-31 2024-03-27T09:34:51.851 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:51.849+0000 7fdfb5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:51.866 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 3 to smithi17031214-31 2024-03-27T09:34:51.875 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 2 to smithi17031214-46 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 4 to smithi17031214-31 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:31: oid 31 updating version 0 to 15 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:31: oid 31 version 15 is already newer than 15 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:31: oid 31 version 15 is already newer than 13 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-27T09:34:51.878 INFO:tasks.rados.rados.0.smithi170.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-03-27T09:34:51.882 INFO:tasks.rados.rados.0.smithi170.stdout:52: finishing write tid 1 to smithi17031214-14 2024-03-27T09:34:51.884 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 3 to smithi17031214-46 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 4 to smithi17031214-46 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:46: oid 46 updating version 0 to 18 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:46: oid 46 version 18 is already newer than 16 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:46: oid 46 version 18 is already newer than 18 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 18 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-27T09:34:51.897 INFO:tasks.rados.rados.0.smithi170.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-03-27T09:34:51.908 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 1 to smithi17031214-37 2024-03-27T09:34:51.915 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 2 to smithi17031214-44 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:31: done (15 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:35: done (14 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:38: done (13 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:42: done (12 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:46: done (11 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:48: done (10 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:54: done (9 left) 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:55: write oid 18 current snap is 0 2024-03-27T09:34:51.919 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 2 to smithi17031214-45 2024-03-27T09:34:51.920 INFO:tasks.rados.rados.0.smithi170.stdout:55: seq_num 53 ranges {490771=686315,1885075=406346} 2024-03-27T09:34:51.927 INFO:tasks.rados.rados.0.smithi170.stdout:55: writing smithi17031214-18 from 490771 to 1177086 tid 1 2024-03-27T09:34:51.936 INFO:tasks.rados.rados.0.smithi170.stdout:55: writing smithi17031214-18 from 1885075 to 2291421 tid 2 2024-03-27T09:34:51.936 INFO:tasks.rados.rados.0.smithi170.stdout:56: write oid 31 current snap is 0 2024-03-27T09:34:51.936 INFO:tasks.rados.rados.0.smithi170.stdout:56: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-03-27T09:34:51.941 INFO:tasks.rados.rados.0.smithi170.stdout:56: writing smithi17031214-31 from 507578 to 985495 tid 1 2024-03-27T09:34:51.948 INFO:tasks.rados.rados.0.smithi170.stdout:56: writing smithi17031214-31 from 1543557 to 2093561 tid 2 2024-03-27T09:34:51.953 INFO:tasks.rados.rados.0.smithi170.stdout:56: writing smithi17031214-31 from 2833665 to 2907578 tid 3 2024-03-27T09:34:51.954 INFO:tasks.rados.rados.0.smithi170.stdout:57: write (excl) oid 22 current snap is 0 2024-03-27T09:34:51.954 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 3 to smithi17031214-44 2024-03-27T09:34:51.954 INFO:tasks.rados.rados.0.smithi170.stdout:57: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-03-27T09:34:51.960 INFO:tasks.rados.rados.0.smithi170.stdout:57: writing smithi17031214-22 from 524385 to 1077551 tid 1 2024-03-27T09:34:51.965 INFO:tasks.rados.rados.0.smithi170.stdout:57: writing smithi17031214-22 from 1602039 to 2012054 tid 2 2024-03-27T09:34:51.971 INFO:tasks.rados.rados.0.smithi170.stdout:57: writing smithi17031214-22 from 2577441 to 2924385 tid 3 2024-03-27T09:34:51.971 INFO:tasks.rados.rados.0.smithi170.stdout:58: read oid 43 snap -1 2024-03-27T09:34:51.972 INFO:tasks.rados.rados.0.smithi170.stdout:58: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-27T09:34:51.972 INFO:tasks.rados.rados.0.smithi170.stdout:59: write (excl) oid 25 current snap is 0 2024-03-27T09:34:51.972 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 1 to smithi17031214-49 2024-03-27T09:34:51.972 INFO:tasks.rados.rados.0.smithi170.stdout:59: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-03-27T09:34:51.978 INFO:tasks.rados.rados.0.smithi170.stdout:59: writing smithi17031214-25 from 541192 to 1169607 tid 1 2024-03-27T09:34:51.984 INFO:tasks.rados.rados.0.smithi170.stdout:59: writing smithi17031214-25 from 1660521 to 2214194 tid 2 2024-03-27T09:34:51.989 INFO:tasks.rados.rados.0.smithi170.stdout:59: writing smithi17031214-25 from 2888511 to 2941192 tid 3 2024-03-27T09:34:51.989 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 4 to smithi17031214-44 2024-03-27T09:34:51.989 INFO:tasks.rados.rados.0.smithi170.stdout:44: oid 44 updating version 0 to 12 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:44: oid 44 version 12 is already newer than 12 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:44: oid 44 version 12 is already newer than 10 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 3 to smithi17031214-45 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 4 to smithi17031214-45 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: oid 45 updating version 0 to 15 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: oid 45 version 15 is already newer than 13 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: oid 45 version 15 is already newer than 15 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-03-27T09:34:51.990 INFO:tasks.rados.rados.0.smithi170.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:52: finishing write tid 2 to smithi17031214-14 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:44: done (13 left) 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:45: done (12 left) 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:53: done (11 left) 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:60: delete oid 21 current snap is 0 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 2 to smithi17031214-37 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:51: finishing write tid 1 to smithi17031214-2 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 3 to smithi17031214-37 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 4 to smithi17031214-37 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: oid 37 updating version 0 to 15 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: oid 37 version 15 is already newer than 15 2024-03-27T09:34:52.105 INFO:tasks.rados.rados.0.smithi170.stdout:37: oid 37 version 15 is already newer than 13 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 1 to smithi17031214-50 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:56: finishing write tid 1 to smithi17031214-31 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 2 to smithi17031214-49 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 3 to smithi17031214-49 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 4 to smithi17031214-49 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 updating version 0 to 10 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 updating version 10 to 12 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 version 12 is already newer than 12 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:57: finishing write tid 1 to smithi17031214-22 2024-03-27T09:34:52.106 INFO:tasks.rados.rados.0.smithi170.stdout:52: finishing write tid 3 to smithi17031214-14 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:52: finishing write tid 4 to smithi17031214-14 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:52: oid 14 updating version 0 to 10 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:52: oid 14 updating version 10 to 12 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:52: oid 14 version 12 is already newer than 11 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: finishing write tid 2 to smithi17031214-2 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: finishing write tid 3 to smithi17031214-2 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: finishing write tid 4 to smithi17031214-2 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: finishing write tid 5 to smithi17031214-2 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: oid 2 updating version 0 to 16 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: oid 2 updating version 16 to 19 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: oid 2 version 19 is already newer than 19 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:51: oid 2 version 19 is already newer than 19 2024-03-27T09:34:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-03-27T09:34:52.108 INFO:tasks.rados.rados.0.smithi170.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2024-03-27T09:34:52.108 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 1 to smithi17031214-41 2024-03-27T09:34:52.108 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 2 to smithi17031214-50 2024-03-27T09:34:52.108 INFO:tasks.rados.rados.0.smithi170.stdout:58: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-27T09:34:52.231 INFO:tasks.rados.rados.0.smithi170.stdout:56: finishing write tid 2 to smithi17031214-31 2024-03-27T09:34:52.231 INFO:tasks.rados.rados.0.smithi170.stdout:56: finishing write tid 3 to smithi17031214-31 2024-03-27T09:34:52.231 INFO:tasks.rados.rados.0.smithi170.stdout:57: finishing write tid 2 to smithi17031214-22 2024-03-27T09:34:52.231 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 3 to smithi17031214-50 2024-03-27T09:34:52.231 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 4 to smithi17031214-50 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 updating version 0 to 15 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 version 15 is already newer than 13 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 version 15 is already newer than 14 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 2 to smithi17031214-41 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:37: done (11 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:49: done (10 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:50: done (9 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:51: done (8 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:52: done (7 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:58: done (6 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:60: done (5 left) 2024-03-27T09:34:52.232 INFO:tasks.rados.rados.0.smithi170.stdout:61: delete oid 38 current snap is 0 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: finishing write tid 4 to smithi17031214-31 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: finishing write tid 5 to smithi17031214-31 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: oid 31 updating version 0 to 19 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: oid 31 version 19 is already newer than 16 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: oid 31 version 19 is already newer than 18 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: oid 31 version 19 is already newer than 18 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 19 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:56: left oid 31 (ObjNum 54 snap 0 seq_num 54) 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:59: finishing write tid 1 to smithi17031214-25 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:57: finishing write tid 3 to smithi17031214-22 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 3 to smithi17031214-41 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 4 to smithi17031214-41 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:41: oid 41 updating version 0 to 16 2024-03-27T09:34:52.233 INFO:tasks.rados.rados.0.smithi170.stdout:41: oid 41 updating version 16 to 18 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:41: oid 41 version 18 is already newer than 18 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:59: finishing write tid 2 to smithi17031214-25 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:59: finishing write tid 3 to smithi17031214-25 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: finishing write tid 4 to smithi17031214-22 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: finishing write tid 5 to smithi17031214-22 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: oid 22 updating version 0 to 10 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: oid 22 updating version 10 to 12 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: oid 22 updating version 12 to 13 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: oid 22 version 13 is already newer than 11 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 13 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:57: left oid 22 (ObjNum 55 snap 0 seq_num 55) 2024-03-27T09:34:52.234 INFO:tasks.rados.rados.0.smithi170.stdout:59: finishing write tid 4 to smithi17031214-25 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: finishing write tid 5 to smithi17031214-25 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: oid 25 updating version 0 to 23 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: oid 25 version 23 is already newer than 20 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: oid 25 version 23 is already newer than 23 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: oid 25 version 23 is already newer than 23 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 23 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:59: left oid 25 (ObjNum 56 snap 0 seq_num 56) 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing write tid 1 to smithi17031214-18 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing write tid 2 to smithi17031214-18 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing write tid 3 to smithi17031214-18 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing write tid 4 to smithi17031214-18 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: oid 18 updating version 0 to 21 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: oid 18 version 21 is already newer than 21 2024-03-27T09:34:52.235 INFO:tasks.rados.rados.0.smithi170.stdout:55: oid 18 version 21 is already newer than 20 2024-03-27T09:34:52.236 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 21 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-27T09:34:52.236 INFO:tasks.rados.rados.0.smithi170.stdout:55: left oid 18 (ObjNum 53 snap 0 seq_num 53) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:41: done (5 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:55: done (4 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:56: done (3 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:57: done (2 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:59: done (1 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:61: done (0 left) 2024-03-27T09:34:52.245 INFO:tasks.rados.rados.0.smithi170.stdout:62: delete oid 27 current snap is 0 2024-03-27T09:34:52.262 INFO:tasks.rados.rados.0.smithi170.stdout:62: done (0 left) 2024-03-27T09:34:52.262 INFO:tasks.rados.rados.0.smithi170.stdout:63: read oid 15 snap -1 2024-03-27T09:34:52.262 INFO:tasks.rados.rados.0.smithi170.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-27T09:34:52.262 INFO:tasks.rados.rados.0.smithi170.stdout:64: snap_create 2024-03-27T09:34:52.278 INFO:tasks.rados.rados.0.smithi170.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-27T09:34:52.572 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:34:52.572 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:34:52.572 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-27T09:34:52.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-27T09:34:52.684 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:52.679+0000 7f2345aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:52.687 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:52.687+0000 7f2345aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:52.704 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:52.707 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:52.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-27T09:34:52.818 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:52.813+0000 7f949dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:52.822 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:52.821+0000 7f949dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:52.833 INFO:tasks.rados.rados.0.smithi170.stdout:63: done (1 left) 2024-03-27T09:34:52.833 INFO:tasks.rados.rados.0.smithi170.stdout:64: done (0 left) 2024-03-27T09:34:52.833 INFO:tasks.rados.rados.0.smithi170.stdout:65: write oid 40 current snap is 1 2024-03-27T09:34:52.833 INFO:tasks.rados.rados.0.smithi170.stdout:65: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-03-27T09:34:52.841 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:52.841 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:52.841 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:52.842 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:52.842 INFO:tasks.rados.rados.0.smithi170.stdout:65: writing smithi17031214-40 from 608420 to 1137831 tid 1 2024-03-27T09:34:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:52.846 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:52.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-27T09:34:52.851 INFO:tasks.rados.rados.0.smithi170.stdout:65: writing smithi17031214-40 from 1778096 to 2339107 tid 2 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:65: writing smithi17031214-40 from 2881850 to 3008420 tid 3 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:66: read oid 48 snap 0 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:67: read oid 28 snap 0 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:68: read oid 13 snap -1 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:69: read oid 43 snap -1 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-27T09:34:52.856 INFO:tasks.rados.rados.0.smithi170.stdout:70: read oid 36 snap -1 2024-03-27T09:34:52.857 INFO:tasks.rados.rados.0.smithi170.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-27T09:34:52.857 INFO:tasks.rados.rados.0.smithi170.stdout:71: read oid 50 snap -1 2024-03-27T09:34:52.857 INFO:tasks.rados.rados.0.smithi170.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-27T09:34:52.857 INFO:tasks.rados.rados.0.smithi170.stdout:72: read oid 8 snap 0 2024-03-27T09:34:52.857 INFO:tasks.rados.rados.0.smithi170.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:34:52.858 INFO:tasks.rados.rados.0.smithi170.stdout:73: read oid 27 snap 0 2024-03-27T09:34:52.858 INFO:tasks.rados.rados.0.smithi170.stdout:73: expect deleted 2024-03-27T09:34:52.858 INFO:tasks.rados.rados.0.smithi170.stdout:74: snap_create 2024-03-27T09:34:52.903 INFO:tasks.rados.rados.0.smithi170.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-27T09:34:53.008 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:53.003+0000 7f9fcde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:53.012 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:53.012+0000 7f9fcde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:53.028 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:53.028 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:53.028 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:53.028 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:53.029 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:53.031 INFO:tasks.rados.rados.0.smithi170.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-27T09:34:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:53.032 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:53.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-27T09:34:53.132 INFO:tasks.rados.rados.0.smithi170.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-27T09:34:53.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:53.180+0000 7f46d635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:53.189 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:53.189+0000 7f46d635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:53.205 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:53.205 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:53.206 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:53.208 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:53.241 INFO:tasks.rados.rados.0.smithi170.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:34:53.321 INFO:tasks.rados.rados.0.smithi170.stdout:65: finishing write tid 1 to smithi17031214-40 2024-03-27T09:34:53.322 INFO:tasks.rados.rados.0.smithi170.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-27T09:34:53.412 INFO:tasks.rados.rados.0.smithi170.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-27T09:34:53.516 INFO:tasks.rados.rados.0.smithi170.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: finishing write tid 2 to smithi17031214-40 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: finishing write tid 3 to smithi17031214-40 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: finishing write tid 4 to smithi17031214-40 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: finishing write tid 5 to smithi17031214-40 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: oid 40 updating version 0 to 23 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: oid 40 updating version 23 to 24 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: oid 40 version 24 is already newer than 23 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: oid 40 version 24 is already newer than 21 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 24 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-03-27T09:34:53.597 INFO:tasks.rados.rados.0.smithi170.stdout:65: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2024-03-27T09:34:53.837 INFO:tasks.rados.rados.0.smithi170.stdout:65: done (9 left) 2024-03-27T09:34:53.837 INFO:tasks.rados.rados.0.smithi170.stdout:66: done (8 left) 2024-03-27T09:34:53.837 INFO:tasks.rados.rados.0.smithi170.stdout:67: done (7 left) 2024-03-27T09:34:53.837 INFO:tasks.rados.rados.0.smithi170.stdout:68: done (6 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:69: done (5 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:70: done (4 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:71: done (3 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:72: done (2 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:73: done (1 left) 2024-03-27T09:34:53.838 INFO:tasks.rados.rados.0.smithi170.stdout:74: done (0 left) 2024-03-27T09:34:53.839 INFO:tasks.rados.rados.0.smithi170.stdout:75: write oid 45 current snap is 2 2024-03-27T09:34:53.839 INFO:tasks.rados.rados.0.smithi170.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2024-03-27T09:34:53.843 INFO:tasks.rados.rados.0.smithi170.stdout:75: writing smithi17031214-45 from 625227 to 1346240 tid 1 2024-03-27T09:34:53.851 INFO:tasks.rados.rados.0.smithi170.stdout:75: writing smithi17031214-45 from 1952931 to 2657600 tid 2 2024-03-27T09:34:53.852 INFO:tasks.rados.rados.0.smithi170.stdout:76: read oid 16 snap -1 2024-03-27T09:34:53.852 INFO:tasks.rados.rados.0.smithi170.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:34:53.852 INFO:tasks.rados.rados.0.smithi170.stdout:77: write oid 46 current snap is 2 2024-03-27T09:34:53.852 INFO:tasks.rados.rados.0.smithi170.stdout:77: seq_num 62 ranges {642034=796262,1895060=564680} 2024-03-27T09:34:53.859 INFO:tasks.rados.rados.0.smithi170.stdout:77: writing smithi17031214-46 from 642034 to 1438296 tid 1 2024-03-27T09:34:53.867 INFO:tasks.rados.rados.0.smithi170.stdout:77: writing smithi17031214-46 from 1895060 to 2459740 tid 2 2024-03-27T09:34:53.867 INFO:tasks.rados.rados.0.smithi170.stdout:78: read oid 26 snap 0 2024-03-27T09:34:53.868 INFO:tasks.rados.rados.0.smithi170.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-27T09:34:53.868 INFO:tasks.rados.rados.0.smithi170.stdout:79: read oid 7 snap -1 2024-03-27T09:34:53.868 INFO:tasks.rados.rados.0.smithi170.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-03-27T09:34:53.868 INFO:tasks.rados.rados.0.smithi170.stdout:80: write oid 44 current snap is 2 2024-03-27T09:34:53.868 INFO:tasks.rados.rados.0.smithi170.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-03-27T09:34:53.873 INFO:tasks.rados.rados.0.smithi170.stdout:80: writing smithi17031214-44 from 658841 to 1130352 tid 1 2024-03-27T09:34:53.880 INFO:tasks.rados.rados.0.smithi170.stdout:80: writing smithi17031214-44 from 1553542 to 2261880 tid 2 2024-03-27T09:34:53.884 INFO:tasks.rados.rados.0.smithi170.stdout:80: writing smithi17031214-44 from 2964119 to 3058841 tid 3 2024-03-27T09:34:53.884 INFO:tasks.rados.rados.0.smithi170.stdout:81: read oid 30 snap -1 2024-03-27T09:34:53.884 INFO:tasks.rados.rados.0.smithi170.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-27T09:34:53.884 INFO:tasks.rados.rados.0.smithi170.stdout:82: snap_remove snap 1 2024-03-27T09:34:54.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:34:54.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:54.327+0000 7f0c4a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:54.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:54.333+0000 7f0c4a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:54.838 INFO:tasks.rados.rados.0.smithi170.stdout:82: done (7 left) 2024-03-27T09:34:54.838 INFO:tasks.rados.rados.0.smithi170.stdout:83: delete oid 49 current snap is 2 2024-03-27T09:34:54.839 INFO:tasks.rados.rados.0.smithi170.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-27T09:34:54.949 INFO:tasks.rados.rados.0.smithi170.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-27T09:34:55.040 INFO:tasks.rados.rados.0.smithi170.stdout:75: finishing write tid 1 to smithi17031214-45 2024-03-27T09:34:55.040 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 1 to smithi17031214-46 2024-03-27T09:34:55.041 INFO:tasks.rados.rados.0.smithi170.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:34:55.125 INFO:tasks.rados.rados.0.smithi170.stdout:80: finishing write tid 1 to smithi17031214-44 2024-03-27T09:34:55.126 INFO:tasks.rados.rados.0.smithi170.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: finishing write tid 2 to smithi17031214-45 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:80: finishing write tid 2 to smithi17031214-44 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:80: finishing write tid 3 to smithi17031214-44 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: finishing write tid 3 to smithi17031214-45 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: finishing write tid 4 to smithi17031214-45 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: oid 45 updating version 0 to 27 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: oid 45 version 27 is already newer than 27 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: oid 45 updating version 27 to 28 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 28 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:75: left oid 45 (ObjNum 61 snap 2 seq_num 61) 2024-03-27T09:34:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 2 to smithi17031214-46 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: finishing write tid 4 to smithi17031214-44 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 3 to smithi17031214-46 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: finishing write tid 5 to smithi17031214-44 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: oid 44 updating version 0 to 20 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: oid 44 version 20 is already newer than 17 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: oid 44 version 20 is already newer than 18 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: oid 44 version 20 is already newer than 19 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 20 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:80: left oid 44 (ObjNum 63 snap 2 seq_num 63) 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 4 to smithi17031214-46 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 46 updating version 0 to 27 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 46 updating version 27 to 28 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 46 version 28 is already newer than 26 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 28 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-03-27T09:34:55.208 INFO:tasks.rados.rados.0.smithi170.stdout:77: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:75: done (7 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:76: done (6 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:77: done (5 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:78: done (4 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:79: done (3 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:80: done (2 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:81: done (1 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:83: done (0 left) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:84: read oid 23 snap -1 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:84: expect deleted 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:85: read oid 26 snap 0 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:85: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:86: write oid 2 current snap is 2 2024-03-27T09:34:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:86: seq_num 65 ranges {692455=622009,2070506=712007} 2024-03-27T09:34:55.213 INFO:tasks.rados.rados.0.smithi170.stdout:86: writing smithi17031214-2 from 692455 to 1314464 tid 1 2024-03-27T09:34:55.222 INFO:tasks.rados.rados.0.smithi170.stdout:86: writing smithi17031214-2 from 2070506 to 2782513 tid 2 2024-03-27T09:34:55.222 INFO:tasks.rados.rados.0.smithi170.stdout:87: snap_create 2024-03-27T09:34:55.289 INFO:tasks.rados.rados.0.smithi170.stdout:85: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: finishing write tid 1 to smithi17031214-2 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: finishing write tid 2 to smithi17031214-2 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: finishing write tid 3 to smithi17031214-2 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: finishing write tid 4 to smithi17031214-2 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: oid 2 updating version 0 to 33 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: oid 2 version 33 is already newer than 33 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: oid 2 updating version 33 to 34 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 34 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-03-27T09:34:55.383 INFO:tasks.rados.rados.0.smithi170.stdout:86: left oid 2 (ObjNum 65 snap 2 seq_num 65) 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:84: done (3 left) 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:85: done (2 left) 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:86: done (1 left) 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:87: done (0 left) 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:88: write oid 6 current snap is 3 2024-03-27T09:34:55.842 INFO:tasks.rados.rados.0.smithi170.stdout:88: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-03-27T09:34:55.849 INFO:tasks.rados.rados.0.smithi170.stdout:88: writing smithi17031214-6 from 709262 to 1406520 tid 1 2024-03-27T09:34:55.855 INFO:tasks.rados.rados.0.smithi170.stdout:88: writing smithi17031214-6 from 2012635 to 2584653 tid 2 2024-03-27T09:34:55.858 INFO:tasks.rados.rados.0.smithi170.stdout:88: writing smithi17031214-6 from 3046388 to 3109262 tid 3 2024-03-27T09:34:55.858 INFO:tasks.rados.rados.0.smithi170.stdout:89: write oid 7 current snap is 3 2024-03-27T09:34:55.858 INFO:tasks.rados.rados.0.smithi170.stdout:89: seq_num 67 ranges {726069=772507,2071117=715676} 2024-03-27T09:34:55.866 INFO:tasks.rados.rados.0.smithi170.stdout:89: writing smithi17031214-7 from 726069 to 1498576 tid 1 2024-03-27T09:34:55.873 INFO:tasks.rados.rados.0.smithi170.stdout:89: writing smithi17031214-7 from 2071117 to 2786793 tid 2 2024-03-27T09:34:55.874 INFO:tasks.rados.rados.0.smithi170.stdout:90: write oid 22 current snap is 3 2024-03-27T09:34:55.874 INFO:tasks.rados.rados.0.smithi170.stdout:90: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-03-27T09:34:55.880 INFO:tasks.rados.rados.0.smithi170.stdout:90: writing smithi17031214-22 from 742876 to 1190632 tid 1 2024-03-27T09:34:55.886 INFO:tasks.rados.rados.0.smithi170.stdout:90: writing smithi17031214-22 from 1729599 to 2305286 tid 2 2024-03-27T09:34:55.890 INFO:tasks.rados.rados.0.smithi170.stdout:90: writing smithi17031214-22 from 3101234 to 3142876 tid 3 2024-03-27T09:34:55.891 INFO:tasks.rados.rados.0.smithi170.stdout:91: delete oid 14 current snap is 3 2024-03-27T09:34:55.918 INFO:tasks.rados.rados.0.smithi170.stdout:88: finishing write tid 1 to smithi17031214-6 2024-03-27T09:34:55.960 INFO:tasks.rados.rados.0.smithi170.stdout:88: finishing write tid 2 to smithi17031214-6 2024-03-27T09:34:55.967 INFO:tasks.rados.rados.0.smithi170.stdout:91: done (3 left) 2024-03-27T09:34:55.967 INFO:tasks.rados.rados.0.smithi170.stdout:92: write (excl) oid 45 current snap is 3 2024-03-27T09:34:55.967 INFO:tasks.rados.rados.0.smithi170.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2024-03-27T09:34:55.976 INFO:tasks.rados.rados.0.smithi170.stdout:92: writing smithi17031214-45 from 776490 to 1491097 tid 1 2024-03-27T09:34:55.983 INFO:tasks.rados.rados.0.smithi170.stdout:92: writing smithi17031214-45 from 2246563 to 2825919 tid 2 2024-03-27T09:34:55.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: write oid 27 current snap is 3 2024-03-27T09:34:55.983 INFO:tasks.rados.rados.0.smithi170.stdout:88: finishing write tid 3 to smithi17031214-6 2024-03-27T09:34:55.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: seq_num 71 ranges {793297=789856,2305045=723014} 2024-03-27T09:34:55.992 INFO:tasks.rados.rados.0.smithi170.stdout:93: writing smithi17031214-27 from 793297 to 1583153 tid 1 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:93: writing smithi17031214-27 from 2305045 to 3028059 tid 2 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 1 to smithi17031214-7 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:94: read oid 14 snap -1 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:94: expect deleted 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:95: read oid 4 snap -1 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:95: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:96: delete oid 49 current snap is 3 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: finishing write tid 4 to smithi17031214-6 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 2 to smithi17031214-7 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: finishing write tid 5 to smithi17031214-6 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: oid 6 updating version 0 to 18 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: oid 6 version 18 is already newer than 17 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: oid 6 version 18 is already newer than 17 2024-03-27T09:34:56.001 INFO:tasks.rados.rados.0.smithi170.stdout:88: oid 6 version 18 is already newer than 15 2024-03-27T09:34:56.002 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 18 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-03-27T09:34:56.002 INFO:tasks.rados.rados.0.smithi170.stdout:88: left oid 6 (ObjNum 66 snap 3 seq_num 66) 2024-03-27T09:34:56.019 INFO:tasks.rados.rados.0.smithi170.stdout:88: done (7 left) 2024-03-27T09:34:56.019 INFO:tasks.rados.rados.0.smithi170.stdout:94: done (6 left) 2024-03-27T09:34:56.019 INFO:tasks.rados.rados.0.smithi170.stdout:96: done (5 left) 2024-03-27T09:34:56.019 INFO:tasks.rados.rados.0.smithi170.stdout:97: snap_remove snap 0 2024-03-27T09:34:56.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:34:56.540 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:56.538+0000 7f184d99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:56.544 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:56.543+0000 7f184d99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:56.844 INFO:tasks.rados.rados.0.smithi170.stdout:97: done (5 left) 2024-03-27T09:34:56.844 INFO:tasks.rados.rados.0.smithi170.stdout:98: write (excl) oid 26 current snap is 3 2024-03-27T09:34:56.844 INFO:tasks.rados.rados.0.smithi170.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-03-27T09:34:56.849 INFO:tasks.rados.rados.0.smithi170.stdout:98: writing smithi17031214-26 from 426911 to 967265 tid 1 2024-03-27T09:34:56.857 INFO:tasks.rados.rados.0.smithi170.stdout:98: writing smithi17031214-26 from 1622009 to 2348692 tid 2 2024-03-27T09:34:56.862 INFO:tasks.rados.rados.0.smithi170.stdout:98: writing smithi17031214-26 from 3121996 to 3226911 tid 3 2024-03-27T09:34:56.863 INFO:tasks.rados.rados.0.smithi170.stdout:99: snap_remove snap 2 2024-03-27T09:34:57.848 INFO:tasks.rados.rados.0.smithi170.stdout:99: done (6 left) 2024-03-27T09:34:57.848 INFO:tasks.rados.rados.0.smithi170.stdout:100: write oid 40 current snap is 3 2024-03-27T09:34:57.848 INFO:tasks.rados.rados.0.smithi170.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-03-27T09:34:57.854 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17031214-40 from 443718 to 1059321 tid 1 2024-03-27T09:34:57.859 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17031214-40 from 1564138 to 2034479 tid 2 2024-03-27T09:34:57.867 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17031214-40 from 2633066 to 3243718 tid 3 2024-03-27T09:34:57.867 INFO:tasks.rados.rados.0.smithi170.stdout:101: read oid 17 snap -1 2024-03-27T09:34:57.867 INFO:tasks.rados.rados.0.smithi170.stdout:101: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:102: read oid 50 snap -1 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:102: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:103: read oid 8 snap -1 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:103: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:104: read oid 1 snap -1 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:104: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:105: read oid 13 snap -1 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:105: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:106: read oid 29 snap -1 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:106: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-27T09:34:57.868 INFO:tasks.rados.rados.0.smithi170.stdout:107: delete oid 11 current snap is 3 2024-03-27T09:34:57.869 INFO:tasks.rados.rados.0.smithi170.stdout:95: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 3 to smithi17031214-7 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 4 to smithi17031214-7 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 7 updating version 0 to 10 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 7 version 10 is already newer than 8 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 7 version 10 is already newer than 9 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 10 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:89: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing write tid 1 to smithi17031214-22 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing write tid 2 to smithi17031214-22 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing write tid 3 to smithi17031214-22 2024-03-27T09:34:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing write tid 4 to smithi17031214-22 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing write tid 5 to smithi17031214-22 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: oid 22 updating version 0 to 23 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: oid 22 version 23 is already newer than 23 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: oid 22 version 23 is already newer than 23 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: oid 22 version 23 is already newer than 23 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 23 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:90: left oid 22 (ObjNum 68 snap 3 seq_num 68) 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing write tid 1 to smithi17031214-45 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing write tid 2 to smithi17031214-45 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing write tid 3 to smithi17031214-45 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing write tid 4 to smithi17031214-45 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: oid 45 updating version 0 to 36 2024-03-27T09:34:57.982 INFO:tasks.rados.rados.0.smithi170.stdout:92: oid 45 updating version 36 to 38 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:92: oid 45 version 38 is already newer than 38 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 38 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:92: left oid 45 (ObjNum 70 snap 3 seq_num 70) 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: finishing write tid 1 to smithi17031214-27 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: finishing write tid 2 to smithi17031214-27 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: finishing write tid 3 to smithi17031214-27 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: finishing write tid 4 to smithi17031214-27 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: oid 27 updating version 0 to 39 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: oid 27 updating version 39 to 41 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: oid 27 version 41 is already newer than 41 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 41 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:93: left oid 27 (ObjNum 71 snap 3 seq_num 71) 2024-03-27T09:34:57.983 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 1 to smithi17031214-26 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 2 to smithi17031214-26 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 3 to smithi17031214-26 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 4 to smithi17031214-26 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 5 to smithi17031214-26 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 26 updating version 0 to 18 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 26 version 18 is already newer than 18 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 26 version 18 is already newer than 16 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 26 updating version 18 to 19 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 19 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:98: left oid 26 (ObjNum 73 snap 3 seq_num 73) 2024-03-27T09:34:57.984 INFO:tasks.rados.rados.0.smithi170.stdout:105: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:34:58.085 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 1 to smithi17031214-40 2024-03-27T09:34:58.085 INFO:tasks.rados.rados.0.smithi170.stdout:104: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:34:58.160 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 2 to smithi17031214-40 2024-03-27T09:34:58.160 INFO:tasks.rados.rados.0.smithi170.stdout:101: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-27T09:34:58.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:34:58.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:34:58.210 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-27T09:34:58.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-27T09:34:58.247 INFO:tasks.rados.rados.0.smithi170.stdout:102: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-27T09:34:58.324 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.318+0000 7fafd0f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.327+0000 7fafd0f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.343 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:58.343 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:58.343 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:58.343 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:58.344 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:58.348 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:58.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-27T09:34:58.357 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 3 to smithi17031214-40 2024-03-27T09:34:58.357 INFO:tasks.rados.rados.0.smithi170.stdout:103: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:34:58.441 INFO:tasks.rados.rados.0.smithi170.stdout:106: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-27T09:34:58.469 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.463+0000 7f00283e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.473 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.472+0000 7f00283e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:58.491 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:58.492 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:58.492 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:58.492 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:58.492 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:58.492 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:58.495 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:58.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 4 to smithi17031214-40 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 5 to smithi17031214-40 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 40 updating version 0 to 38 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 40 version 38 is already newer than 37 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 40 version 38 is already newer than 37 2024-03-27T09:34:58.543 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 40 version 38 is already newer than 35 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 38 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:100: left oid 40 (ObjNum 74 snap 3 seq_num 74) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:89: done (13 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:90: done (12 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:92: done (11 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:93: done (10 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:95: done (9 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:98: done (8 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:100: done (7 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:101: done (6 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:102: done (5 left) 2024-03-27T09:34:58.544 INFO:tasks.rados.rados.0.smithi170.stdout:103: done (4 left) 2024-03-27T09:34:58.545 INFO:tasks.rados.rados.0.smithi170.stdout:104: done (3 left) 2024-03-27T09:34:58.545 INFO:tasks.rados.rados.0.smithi170.stdout:105: done (2 left) 2024-03-27T09:34:58.545 INFO:tasks.rados.rados.0.smithi170.stdout:106: done (1 left) 2024-03-27T09:34:58.545 INFO:tasks.rados.rados.0.smithi170.stdout:107: done (0 left) 2024-03-27T09:34:58.545 INFO:tasks.rados.rados.0.smithi170.stdout:108: snap_create 2024-03-27T09:34:58.605 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.600+0000 7fe15519d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.609 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.608+0000 7fe15519d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.625 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:58.626 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:58.629 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:58.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-27T09:34:58.644 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:34:58.742 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.737+0000 7fc0596ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.747 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.747+0000 7fc0596ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:34:58.765 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:34:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:34:58.769 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:34:58.784 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.782+0000 7ffbea46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:34:58.787+0000 7ffbea46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:34:58.850 INFO:tasks.rados.rados.0.smithi170.stdout:108: done (0 left) 2024-03-27T09:34:58.851 INFO:tasks.rados.rados.0.smithi170.stdout:109: delete oid 42 current snap is 4 2024-03-27T09:34:58.905 INFO:tasks.rados.rados.0.smithi170.stdout:109: done (0 left) 2024-03-27T09:34:58.905 INFO:tasks.rados.rados.0.smithi170.stdout:110: read oid 48 snap 3 2024-03-27T09:34:58.905 INFO:tasks.rados.rados.0.smithi170.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-27T09:34:58.905 INFO:tasks.rados.rados.0.smithi170.stdout:111: snap_create 2024-03-27T09:34:58.929 INFO:tasks.rados.rados.0.smithi170.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-27T09:34:59.854 INFO:tasks.rados.rados.0.smithi170.stdout:110: done (1 left) 2024-03-27T09:34:59.855 INFO:tasks.rados.rados.0.smithi170.stdout:111: done (0 left) 2024-03-27T09:34:59.855 INFO:tasks.rados.rados.0.smithi170.stdout:112: snap_create 2024-03-27T09:35:00.858 INFO:tasks.rados.rados.0.smithi170.stdout:112: done (0 left) 2024-03-27T09:35:00.859 INFO:tasks.rados.rados.0.smithi170.stdout:113: read oid 35 snap -1 2024-03-27T09:35:00.859 INFO:tasks.rados.rados.0.smithi170.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-27T09:35:00.859 INFO:tasks.rados.rados.0.smithi170.stdout:114: read oid 9 snap -1 2024-03-27T09:35:00.859 INFO:tasks.rados.rados.0.smithi170.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-27T09:35:00.859 INFO:tasks.rados.rados.0.smithi170.stdout:115: delete oid 26 current snap is 6 2024-03-27T09:35:00.911 INFO:tasks.rados.rados.0.smithi170.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-27T09:35:00.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:01.019 INFO:tasks.rados.rados.0.smithi170.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-27T09:35:01.045 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:01.042+0000 7f187015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:01.048 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:01.048+0000 7f187015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:01.115 INFO:tasks.rados.rados.0.smithi170.stdout:113: done (2 left) 2024-03-27T09:35:01.115 INFO:tasks.rados.rados.0.smithi170.stdout:114: done (1 left) 2024-03-27T09:35:01.115 INFO:tasks.rados.rados.0.smithi170.stdout:115: done (0 left) 2024-03-27T09:35:01.115 INFO:tasks.rados.rados.0.smithi170.stdout:116: write oid 28 current snap is 6 2024-03-27T09:35:01.115 INFO:tasks.rados.rados.0.smithi170.stdout:116: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-03-27T09:35:01.121 INFO:tasks.rados.rados.0.smithi170.stdout:116: writing smithi17031214-28 from 510946 to 1143898 tid 1 2024-03-27T09:35:01.127 INFO:tasks.rados.rados.0.smithi170.stdout:116: writing smithi17031214-28 from 1798066 to 2275745 tid 2 2024-03-27T09:35:01.134 INFO:tasks.rados.rados.0.smithi170.stdout:116: writing smithi17031214-28 from 2742758 to 3199975 tid 3 2024-03-27T09:35:01.135 INFO:tasks.rados.rados.0.smithi170.stdout:117: write oid 50 current snap is 6 2024-03-27T09:35:01.135 INFO:tasks.rados.rados.0.smithi170.stdout:117: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-03-27T09:35:01.145 INFO:tasks.rados.rados.0.smithi170.stdout:117: writing smithi17031214-50 from 527753 to 1235954 tid 1 2024-03-27T09:35:01.154 INFO:tasks.rados.rados.0.smithi170.stdout:117: writing smithi17031214-50 from 1856548 to 2594238 tid 2 2024-03-27T09:35:01.158 INFO:tasks.rados.rados.0.smithi170.stdout:117: writing smithi17031214-50 from 3286534 to 3327753 tid 3 2024-03-27T09:35:01.158 INFO:tasks.rados.rados.0.smithi170.stdout:118: write oid 49 current snap is 6 2024-03-27T09:35:01.159 INFO:tasks.rados.rados.0.smithi170.stdout:118: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-03-27T09:35:01.166 INFO:tasks.rados.rados.0.smithi170.stdout:118: writing smithi17031214-49 from 544560 to 1328010 tid 1 2024-03-27T09:35:01.175 INFO:tasks.rados.rados.0.smithi170.stdout:118: writing smithi17031214-49 from 1915030 to 2396378 tid 2 2024-03-27T09:35:01.181 INFO:tasks.rados.rados.0.smithi170.stdout:118: writing smithi17031214-49 from 2797604 to 3277365 tid 3 2024-03-27T09:35:01.181 INFO:tasks.rados.rados.0.smithi170.stdout:119: write (excl) oid 26 current snap is 6 2024-03-27T09:35:01.181 INFO:tasks.rados.rados.0.smithi170.stdout:116: finishing write tid 1 to smithi17031214-28 2024-03-27T09:35:01.181 INFO:tasks.rados.rados.0.smithi170.stdout:119: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-03-27T09:35:01.189 INFO:tasks.rados.rados.0.smithi170.stdout:119: writing smithi17031214-26 from 561367 to 1020066 tid 1 2024-03-27T09:35:01.199 INFO:tasks.rados.rados.0.smithi170.stdout:119: writing smithi17031214-26 from 1573512 to 2314871 tid 2 2024-03-27T09:35:01.208 INFO:tasks.rados.rados.0.smithi170.stdout:119: writing smithi17031214-26 from 2941380 to 3361367 tid 3 2024-03-27T09:35:01.208 INFO:tasks.rados.rados.0.smithi170.stdout:120: write (excl) oid 44 current snap is 6 2024-03-27T09:35:01.208 INFO:tasks.rados.rados.0.smithi170.stdout:116: finishing write tid 2 to smithi17031214-28 2024-03-27T09:35:01.208 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing write tid 1 to smithi17031214-49 2024-03-27T09:35:01.208 INFO:tasks.rados.rados.0.smithi170.stdout:120: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-03-27T09:35:01.215 INFO:tasks.rados.rados.0.smithi170.stdout:120: writing smithi17031214-44 from 578174 to 1112122 tid 1 2024-03-27T09:35:01.220 INFO:tasks.rados.rados.0.smithi170.stdout:120: writing smithi17031214-44 from 1515641 to 2000658 tid 2 2024-03-27T09:35:01.227 INFO:tasks.rados.rados.0.smithi170.stdout:120: writing smithi17031214-44 from 2736097 to 3238402 tid 3 2024-03-27T09:35:01.227 INFO:tasks.rados.rados.0.smithi170.stdout:121: read oid 22 snap -1 2024-03-27T09:35:01.228 INFO:tasks.rados.rados.0.smithi170.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-27T09:35:01.228 INFO:tasks.rados.rados.0.smithi170.stdout:122: snap_remove snap 5 2024-03-27T09:35:01.862 INFO:tasks.rados.rados.0.smithi170.stdout:122: done (6 left) 2024-03-27T09:35:01.862 INFO:tasks.rados.rados.0.smithi170.stdout:123: read oid 37 snap -1 2024-03-27T09:35:01.862 INFO:tasks.rados.rados.0.smithi170.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-27T09:35:01.862 INFO:tasks.rados.rados.0.smithi170.stdout:124: snap_remove snap 4 2024-03-27T09:35:02.866 INFO:tasks.rados.rados.0.smithi170.stdout:124: done (7 left) 2024-03-27T09:35:02.866 INFO:tasks.rados.rados.0.smithi170.stdout:125: snap_remove snap 3 2024-03-27T09:35:03.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:03.291 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:03.289+0000 7f8ee8ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:03.294 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:03.294+0000 7f8ee8ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:03.770 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:03.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-27T09:35:03.869 INFO:tasks.rados.rados.0.smithi170.stdout:125: done (7 left) 2024-03-27T09:35:03.869 INFO:tasks.rados.rados.0.smithi170.stdout:126: write oid 36 current snap is 6 2024-03-27T09:35:03.870 INFO:tasks.rados.rados.0.smithi170.stdout:116: finishing write tid 3 to smithi17031214-28 2024-03-27T09:35:03.870 INFO:tasks.rados.rados.0.smithi170.stdout:126: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-03-27T09:35:03.876 INFO:tasks.rados.rados.0.smithi170.stdout:126: writing smithi17031214-36 from 594981 to 1204178 tid 1 2024-03-27T09:35:03.884 INFO:tasks.rados.rados.0.smithi170.stdout:126: writing smithi17031214-36 from 1974123 to 2719151 tid 2 2024-03-27T09:35:03.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:03.879+0000 7f6b2af1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:03.889 INFO:tasks.rados.rados.0.smithi170.stdout:126: writing smithi17031214-36 from 3279873 to 3394981 tid 3 2024-03-27T09:35:03.889 INFO:tasks.rados.rados.0.smithi170.stdout:127: write (excl) oid 35 current snap is 6 2024-03-27T09:35:03.889 INFO:tasks.rados.rados.0.smithi170.stdout:119: finishing write tid 1 to smithi17031214-26 2024-03-27T09:35:03.889 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing write tid 2 to smithi17031214-49 2024-03-27T09:35:03.889 INFO:tasks.rados.rados.0.smithi170.stdout:127: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-03-27T09:35:03.890 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:03.889+0000 7f6b2af1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:03.894 INFO:tasks.rados.rados.0.smithi170.stdout:127: writing smithi17031214-35 from 611788 to 1012587 tid 1 2024-03-27T09:35:03.901 INFO:tasks.rados.rados.0.smithi170.stdout:127: writing smithi17031214-35 from 1748958 to 2237644 tid 2 2024-03-27T09:35:03.909 INFO:tasks.rados.rados.0.smithi170.stdout:127: writing smithi17031214-35 from 2907296 to 3411788 tid 3 2024-03-27T09:35:03.909 INFO:tasks.rados.rados.0.smithi170.stdout:128: write oid 2 current snap is 6 2024-03-27T09:35:03.909 INFO:tasks.rados.rados.0.smithi170.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-27T09:35:04.137 INFO:tasks.rados.rados.0.smithi170.stdout:119: finishing write tid 2 to smithi17031214-26 2024-03-27T09:35:04.137 INFO:tasks.rados.rados.0.smithi170.stdout:116: finishing write tid 4 to smithi17031214-28 2024-03-27T09:35:04.137 INFO:tasks.rados.rados.0.smithi170.stdout:128: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-03-27T09:35:04.143 INFO:tasks.rados.rados.0.smithi170.stdout:128: writing smithi17031214-2 from 628595 to 1104643 tid 1 2024-03-27T09:35:04.151 INFO:tasks.rados.rados.0.smithi170.stdout:128: writing smithi17031214-2 from 1807440 to 2439784 tid 2 2024-03-27T09:35:04.157 INFO:tasks.rados.rados.0.smithi170.stdout:128: writing smithi17031214-2 from 2934719 to 3387193 tid 3 2024-03-27T09:35:04.157 INFO:tasks.rados.rados.0.smithi170.stdout:121: done (9 left) 2024-03-27T09:35:04.157 INFO:tasks.rados.rados.0.smithi170.stdout:129: read oid 14 snap -1 2024-03-27T09:35:04.157 INFO:tasks.rados.rados.0.smithi170.stdout:129: expect deleted 2024-03-27T09:35:04.158 INFO:tasks.rados.rados.0.smithi170.stdout:130: read oid 13 snap -1 2024-03-27T09:35:04.158 INFO:tasks.rados.rados.0.smithi170.stdout:130: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:35:04.158 INFO:tasks.rados.rados.0.smithi170.stdout:131: write oid 27 current snap is 6 2024-03-27T09:35:04.158 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing write tid 3 to smithi17031214-49 2024-03-27T09:35:04.158 INFO:tasks.rados.rados.0.smithi170.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-03-27T09:35:04.164 INFO:tasks.rados.rados.0.smithi170.stdout:131: writing smithi17031214-27 from 645402 to 1196699 tid 1 2024-03-27T09:35:04.172 INFO:tasks.rados.rados.0.smithi170.stdout:131: writing smithi17031214-27 from 1749569 to 2241924 tid 2 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:131: writing smithi17031214-27 from 2845789 to 3445402 tid 3 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:132: read oid 21 snap -1 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:132: expect deleted 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:133: write oid 3 current snap is 6 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:116: finishing write tid 5 to smithi17031214-28 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:116: oid 28 updating version 0 to 26 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:116: oid 28 version 26 is already newer than 25 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:116: oid 28 version 26 is already newer than 24 2024-03-27T09:35:04.183 INFO:tasks.rados.rados.0.smithi170.stdout:116: oid 28 version 26 is already newer than 23 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 26 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:116: left oid 28 (ObjNum 78 snap 6 seq_num 78) 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:119: finishing write tid 3 to smithi17031214-26 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing write tid 4 to smithi17031214-49 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing write tid 5 to smithi17031214-49 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: oid 49 updating version 0 to 19 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: oid 49 version 19 is already newer than 18 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: oid 49 updating version 19 to 20 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: oid 49 version 20 is already newer than 17 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 20 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:118: left oid 49 (ObjNum 80 snap 6 seq_num 80) 2024-03-27T09:35:04.184 INFO:tasks.rados.rados.0.smithi170.stdout:117: finishing write tid 1 to smithi17031214-50 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: finishing write tid 4 to smithi17031214-26 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: finishing write tid 5 to smithi17031214-26 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: oid 26 updating version 0 to 25 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: oid 26 updating version 25 to 26 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: oid 26 updating version 26 to 27 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: oid 26 version 27 is already newer than 24 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 27 (ObjNum 81 snap 6 seq_num 81) dirty exists 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:119: left oid 26 (ObjNum 81 snap 6 seq_num 81) 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:117: finishing write tid 2 to smithi17031214-50 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:117: finishing write tid 3 to smithi17031214-50 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:117: finishing write tid 4 to smithi17031214-50 2024-03-27T09:35:04.185 INFO:tasks.rados.rados.0.smithi170.stdout:117: finishing write tid 5 to smithi17031214-50 2024-03-27T09:35:04.186 INFO:tasks.rados.rados.0.smithi170.stdout:117: oid 50 updating version 0 to 28 2024-03-27T09:35:04.186 INFO:tasks.rados.rados.0.smithi170.stdout:117: oid 50 version 28 is already newer than 27 2024-03-27T09:35:04.186 INFO:tasks.rados.rados.0.smithi170.stdout:117: oid 50 updating version 28 to 29 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:117: oid 50 version 29 is already newer than 26 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 29 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:117: left oid 50 (ObjNum 79 snap 6 seq_num 79) 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: finishing write tid 1 to smithi17031214-44 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: finishing write tid 2 to smithi17031214-44 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: finishing write tid 3 to smithi17031214-44 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: finishing write tid 4 to smithi17031214-44 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: finishing write tid 5 to smithi17031214-44 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: oid 44 updating version 0 to 34 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: oid 44 version 34 is already newer than 34 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: oid 44 version 34 is already newer than 34 2024-03-27T09:35:04.187 INFO:tasks.rados.rados.0.smithi170.stdout:120: oid 44 version 34 is already newer than 31 2024-03-27T09:35:04.188 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 34 (ObjNum 82 snap 6 seq_num 82) dirty exists 2024-03-27T09:35:04.188 INFO:tasks.rados.rados.0.smithi170.stdout:120: left oid 44 (ObjNum 82 snap 6 seq_num 82) 2024-03-27T09:35:04.188 INFO:tasks.rados.rados.0.smithi170.stdout:123: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing write tid 1 to smithi17031214-35 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing write tid 2 to smithi17031214-35 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:126: finishing write tid 1 to smithi17031214-36 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing write tid 3 to smithi17031214-35 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing write tid 4 to smithi17031214-35 2024-03-27T09:35:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing write tid 5 to smithi17031214-35 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:127: oid 35 updating version 0 to 34 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:127: oid 35 updating version 34 to 35 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:127: oid 35 version 35 is already newer than 33 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:127: oid 35 updating version 35 to 36 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 36 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:127: left oid 35 (ObjNum 84 snap 6 seq_num 84) 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:126: finishing write tid 2 to smithi17031214-36 2024-03-27T09:35:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:133: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-03-27T09:35:04.300 INFO:tasks.rados.rados.0.smithi170.stdout:133: writing smithi17031214-3 from 662209 to 1288755 tid 1 2024-03-27T09:35:04.307 INFO:tasks.rados.rados.0.smithi170.stdout:133: writing smithi17031214-3 from 1808051 to 2444064 tid 2 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:133: writing smithi17031214-3 from 2873212 to 3462209 tid 3 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:116: done (13 left) 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:117: done (12 left) 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:118: done (11 left) 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:119: done (10 left) 2024-03-27T09:35:04.318 INFO:tasks.rados.rados.0.smithi170.stdout:120: done (9 left) 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:123: done (8 left) 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:127: done (7 left) 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:134: snap_create 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: finishing write tid 3 to smithi17031214-36 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: finishing write tid 4 to smithi17031214-36 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: finishing write tid 5 to smithi17031214-36 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: oid 36 updating version 0 to 42 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: oid 36 version 42 is already newer than 42 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: oid 36 version 42 is already newer than 40 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: oid 36 version 42 is already newer than 39 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 42 (ObjNum 83 snap 6 seq_num 83) dirty exists 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:126: left oid 36 (ObjNum 83 snap 6 seq_num 83) 2024-03-27T09:35:04.319 INFO:tasks.rados.rados.0.smithi170.stdout:130: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: finishing write tid 1 to smithi17031214-2 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: finishing write tid 2 to smithi17031214-2 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: finishing write tid 3 to smithi17031214-2 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: finishing write tid 4 to smithi17031214-2 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: finishing write tid 5 to smithi17031214-2 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: oid 2 updating version 0 to 48 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: oid 2 updating version 48 to 50 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: oid 2 updating version 50 to 51 2024-03-27T09:35:04.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: oid 2 version 51 is already newer than 50 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 51 (ObjNum 85 snap 6 seq_num 85) dirty exists 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:128: left oid 2 (ObjNum 85 snap 6 seq_num 85) 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:133: finishing write tid 1 to smithi17031214-3 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: finishing write tid 1 to smithi17031214-27 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: finishing write tid 2 to smithi17031214-27 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:133: finishing write tid 2 to smithi17031214-3 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: finishing write tid 3 to smithi17031214-27 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: finishing write tid 4 to smithi17031214-27 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: finishing write tid 5 to smithi17031214-27 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: oid 27 updating version 0 to 55 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: oid 27 version 55 is already newer than 55 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: oid 27 version 55 is already newer than 55 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: oid 27 version 55 is already newer than 55 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 55 (ObjNum 86 snap 6 seq_num 86) dirty exists 2024-03-27T09:35:04.427 INFO:tasks.rados.rados.0.smithi170.stdout:131: left oid 27 (ObjNum 86 snap 6 seq_num 86) 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: finishing write tid 3 to smithi17031214-3 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: finishing write tid 4 to smithi17031214-3 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: finishing write tid 5 to smithi17031214-3 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: oid 3 updating version 0 to 21 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: oid 3 updating version 21 to 22 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: oid 3 updating version 22 to 23 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: oid 3 updating version 23 to 24 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 24 (ObjNum 87 snap 6 seq_num 87) dirty exists 2024-03-27T09:35:04.428 INFO:tasks.rados.rados.0.smithi170.stdout:133: left oid 3 (ObjNum 87 snap 6 seq_num 87) 2024-03-27T09:35:04.868 INFO:teuthology.orchestra.run.smithi170.stderr:reweighted osd.0 to 1 (10000) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:126: done (7 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:128: done (6 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:129: done (5 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:130: done (4 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:131: done (3 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:132: done (2 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:133: done (1 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:134: done (0 left) 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:135: write (excl) oid 42 current snap is 7 2024-03-27T09:35:04.872 INFO:tasks.rados.rados.0.smithi170.stdout:135: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-03-27T09:35:04.879 INFO:tasks.rados.rados.0.smithi170.stdout:135: writing smithi17031214-42 from 679016 to 1380811 tid 1 2024-03-27T09:35:04.885 INFO:tasks.rados.rados.0.smithi170.stdout:135: writing smithi17031214-42 from 1866533 to 2362557 tid 2 2024-03-27T09:35:04.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-27T09:35:04.891 INFO:tasks.rados.rados.0.smithi170.stdout:135: writing smithi17031214-42 from 2900635 to 3303278 tid 3 2024-03-27T09:35:04.891 INFO:tasks.rados.rados.0.smithi170.stdout:136: snap_remove snap 6 2024-03-27T09:35:05.031 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:05.025+0000 7fb54ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:05.036 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:05.035+0000 7fb54ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:05.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:05.506 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:05.504+0000 7f7f89e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:05.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:05.509+0000 7f7f89e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:05.875 INFO:teuthology.orchestra.run.smithi170.stderr:reweighted osd.1 to 1 (10000) 2024-03-27T09:35:05.879 INFO:tasks.rados.rados.0.smithi170.stdout:136: done (1 left) 2024-03-27T09:35:05.879 INFO:tasks.rados.rados.0.smithi170.stdout:137: read oid 2 snap -1 2024-03-27T09:35:05.879 INFO:tasks.rados.rados.0.smithi170.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-03-27T09:35:05.879 INFO:tasks.rados.rados.0.smithi170.stdout:138: read oid 16 snap -1 2024-03-27T09:35:05.879 INFO:tasks.rados.rados.0.smithi170.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:139: read oid 32 snap -1 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:140: read oid 22 snap -1 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:141: write oid 17 current snap is 7 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:135: finishing write tid 1 to smithi17031214-42 2024-03-27T09:35:05.880 INFO:tasks.rados.rados.0.smithi170.stdout:141: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-03-27T09:35:05.891 INFO:tasks.rados.rados.0.smithi170.stdout:141: writing smithi17031214-17 from 695823 to 1472867 tid 1 2024-03-27T09:35:05.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-27T09:35:05.900 INFO:tasks.rados.rados.0.smithi170.stdout:141: writing smithi17031214-17 from 1925015 to 2564697 tid 2 2024-03-27T09:35:05.906 INFO:tasks.rados.rados.0.smithi170.stdout:141: writing smithi17031214-17 from 3328058 to 3495823 tid 3 2024-03-27T09:35:05.906 INFO:tasks.rados.rados.0.smithi170.stdout:142: write (excl) oid 36 current snap is 7 2024-03-27T09:35:05.906 INFO:tasks.rados.rados.0.smithi170.stdout:142: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-03-27T09:35:05.913 INFO:tasks.rados.rados.0.smithi170.stdout:142: writing smithi17031214-36 from 712630 to 1164923 tid 1 2024-03-27T09:35:05.920 INFO:tasks.rados.rados.0.smithi170.stdout:142: writing smithi17031214-36 from 1583497 to 2083190 tid 2 2024-03-27T09:35:05.925 INFO:tasks.rados.rados.0.smithi170.stdout:142: writing smithi17031214-36 from 2555481 to 2980668 tid 3 2024-03-27T09:35:05.926 INFO:tasks.rados.rados.0.smithi170.stdout:143: write (excl) oid 9 current snap is 7 2024-03-27T09:35:05.926 INFO:tasks.rados.rados.0.smithi170.stdout:135: finishing write tid 2 to smithi17031214-42 2024-03-27T09:35:05.926 INFO:tasks.rados.rados.0.smithi170.stdout:135: finishing write tid 3 to smithi17031214-42 2024-03-27T09:35:05.926 INFO:tasks.rados.rados.0.smithi170.stdout:143: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-03-27T09:35:05.934 INFO:tasks.rados.rados.0.smithi170.stdout:143: writing smithi17031214-9 from 729437 to 1256979 tid 1 2024-03-27T09:35:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:143: writing smithi17031214-9 from 1925626 to 2568977 tid 2 2024-03-27T09:35:05.951 INFO:tasks.rados.rados.0.smithi170.stdout:143: writing smithi17031214-9 from 3266551 to 3529437 tid 3 2024-03-27T09:35:05.951 INFO:tasks.rados.rados.0.smithi170.stdout:144: write oid 11 current snap is 7 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: finishing write tid 4 to smithi17031214-42 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: finishing write tid 5 to smithi17031214-42 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: oid 42 updating version 0 to 45 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: oid 42 updating version 45 to 46 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: oid 42 version 46 is already newer than 45 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: oid 42 version 46 is already newer than 43 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 46 (ObjNum 88 snap 7 seq_num 88) dirty exists 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:135: left oid 42 (ObjNum 88 snap 7 seq_num 88) 2024-03-27T09:35:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:144: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-03-27T09:35:05.963 INFO:tasks.rados.rados.0.smithi170.stdout:144: writing smithi17031214-11 from 746244 to 1465388 tid 1 2024-03-27T09:35:05.971 INFO:tasks.rados.rados.0.smithi170.stdout:144: writing smithi17031214-11 from 2100461 to 2603823 tid 2 2024-03-27T09:35:05.977 INFO:tasks.rados.rados.0.smithi170.stdout:144: writing smithi17031214-11 from 3126680 to 3546244 tid 3 2024-03-27T09:35:05.977 INFO:tasks.rados.rados.0.smithi170.stdout:135: done (8 left) 2024-03-27T09:35:05.978 INFO:tasks.rados.rados.0.smithi170.stdout:145: write oid 14 current snap is 7 2024-03-27T09:35:05.978 INFO:tasks.rados.rados.0.smithi170.stdout:145: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-03-27T09:35:05.988 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17031214-14 from 763051 to 1557444 tid 1 2024-03-27T09:35:05.998 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17031214-14 from 2158943 to 2805963 tid 2 2024-03-27T09:35:06.003 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17031214-14 from 3437750 to 3563051 tid 3 2024-03-27T09:35:06.003 INFO:tasks.rados.rados.0.smithi170.stdout:146: write oid 31 current snap is 7 2024-03-27T09:35:06.003 INFO:tasks.rados.rados.0.smithi170.stdout:146: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-03-27T09:35:06.011 INFO:tasks.rados.rados.0.smithi170.stdout:146: writing smithi17031214-31 from 779858 to 1249500 tid 1 2024-03-27T09:35:06.018 INFO:tasks.rados.rados.0.smithi170.stdout:146: writing smithi17031214-31 from 1817425 to 2324456 tid 2 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:146: writing smithi17031214-31 from 2781526 to 3368154 tid 3 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:147: read oid 33 snap -1 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:147: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:148: read oid 46 snap -1 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:148: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-27T09:35:06.025 INFO:tasks.rados.rados.0.smithi170.stdout:149: delete oid 48 current snap is 7 2024-03-27T09:35:06.026 INFO:tasks.rados.rados.0.smithi170.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:06.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:06.077+0000 7f4bcea06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:06.097 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:06.097+0000 7f4bcea06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:06.137 INFO:tasks.rados.rados.0.smithi170.stdout:141: finishing write tid 1 to smithi17031214-17 2024-03-27T09:35:06.138 INFO:tasks.rados.rados.0.smithi170.stdout:138: done (12 left) 2024-03-27T09:35:06.138 INFO:tasks.rados.rados.0.smithi170.stdout:149: done (11 left) 2024-03-27T09:35:06.138 INFO:tasks.rados.rados.0.smithi170.stdout:150: write (excl) oid 18 current snap is 7 2024-03-27T09:35:06.138 INFO:tasks.rados.rados.0.smithi170.stdout:150: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-03-27T09:35:06.146 INFO:tasks.rados.rados.0.smithi170.stdout:150: writing smithi17031214-18 from 413472 to 1033612 tid 1 2024-03-27T09:35:06.156 INFO:tasks.rados.rados.0.smithi170.stdout:150: writing smithi17031214-18 from 1818036 to 2328736 tid 2 2024-03-27T09:35:06.164 INFO:tasks.rados.rados.0.smithi170.stdout:150: writing smithi17031214-18 from 3120019 to 3613472 tid 3 2024-03-27T09:35:06.165 INFO:tasks.rados.rados.0.smithi170.stdout:151: delete oid 19 current snap is 7 2024-03-27T09:35:06.165 INFO:tasks.rados.rados.0.smithi170.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-27T09:35:06.393 INFO:tasks.rados.rados.0.smithi170.stdout:143: finishing write tid 1 to smithi17031214-9 2024-03-27T09:35:06.394 INFO:tasks.rados.rados.0.smithi170.stdout:140: done (12 left) 2024-03-27T09:35:06.394 INFO:tasks.rados.rados.0.smithi170.stdout:151: done (11 left) 2024-03-27T09:35:06.394 INFO:tasks.rados.rados.0.smithi170.stdout:152: write (excl) oid 40 current snap is 7 2024-03-27T09:35:06.394 INFO:tasks.rados.rados.0.smithi170.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-03-27T09:35:06.615 INFO:tasks.rados.rados.0.smithi170.stdout:139: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:06.708 INFO:tasks.rados.rados.0.smithi170.stdout:144: finishing write tid 1 to smithi17031214-11 2024-03-27T09:35:06.708 INFO:tasks.rados.rados.0.smithi170.stdout:143: finishing write tid 2 to smithi17031214-9 2024-03-27T09:35:06.708 INFO:tasks.rados.rados.0.smithi170.stdout:143: finishing write tid 3 to smithi17031214-9 2024-03-27T09:35:06.708 INFO:tasks.rados.rados.0.smithi170.stdout:144: finishing write tid 2 to smithi17031214-11 2024-03-27T09:35:06.708 INFO:tasks.rados.rados.0.smithi170.stdout:141: finishing write tid 2 to smithi17031214-17 2024-03-27T09:35:06.709 INFO:tasks.rados.rados.0.smithi170.stdout:144: finishing write tid 3 to smithi17031214-11 2024-03-27T09:35:06.709 INFO:tasks.rados.rados.0.smithi170.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-03-27T09:35:06.715 INFO:tasks.rados.rados.0.smithi170.stdout:152: writing smithi17031214-40 from 447086 to 1217724 tid 1 2024-03-27T09:35:06.723 INFO:tasks.rados.rados.0.smithi170.stdout:152: writing smithi17031214-40 from 1935000 to 2733016 tid 2 2024-03-27T09:35:06.729 INFO:tasks.rados.rados.0.smithi170.stdout:152: writing smithi17031214-40 from 3458512 to 3647086 tid 3 2024-03-27T09:35:06.729 INFO:tasks.rados.rados.0.smithi170.stdout:137: done (11 left) 2024-03-27T09:35:06.729 INFO:tasks.rados.rados.0.smithi170.stdout:139: done (10 left) 2024-03-27T09:35:06.729 INFO:tasks.rados.rados.0.smithi170.stdout:153: read oid 16 snap -1 2024-03-27T09:35:06.730 INFO:tasks.rados.rados.0.smithi170.stdout:153: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:06.730 INFO:tasks.rados.rados.0.smithi170.stdout:154: snap_create 2024-03-27T09:35:06.730 INFO:tasks.rados.rados.0.smithi170.stdout:147: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 1 to smithi17031214-14 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:143: finishing write tid 4 to smithi17031214-9 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:143: finishing write tid 5 to smithi17031214-9 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:143: oid 9 updating version 0 to 48 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:143: oid 9 version 48 is already newer than 47 2024-03-27T09:35:06.845 INFO:tasks.rados.rados.0.smithi170.stdout:143: oid 9 updating version 48 to 50 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:143: oid 9 version 50 is already newer than 49 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 50 (ObjNum 91 snap 7 seq_num 91) dirty exists 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:143: left oid 9 (ObjNum 91 snap 7 seq_num 91) 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: finishing write tid 3 to smithi17031214-17 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:144: finishing write tid 4 to smithi17031214-11 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: finishing write tid 4 to smithi17031214-17 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: finishing write tid 5 to smithi17031214-17 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: oid 17 updating version 0 to 46 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: oid 17 version 46 is already newer than 44 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: oid 17 version 46 is already newer than 46 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: oid 17 version 46 is already newer than 43 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 46 (ObjNum 89 snap 7 seq_num 89) dirty exists 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:141: left oid 17 (ObjNum 89 snap 7 seq_num 89) 2024-03-27T09:35:06.846 INFO:tasks.rados.rados.0.smithi170.stdout:144: finishing write tid 5 to smithi17031214-11 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:144: oid 11 updating version 0 to 22 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:144: oid 11 updating version 22 to 24 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:144: oid 11 version 24 is already newer than 21 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:144: oid 11 version 24 is already newer than 23 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 24 (ObjNum 92 snap 7 seq_num 92) dirty exists 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:144: left oid 11 (ObjNum 92 snap 7 seq_num 92) 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 2 to smithi17031214-14 2024-03-27T09:35:06.847 INFO:tasks.rados.rados.0.smithi170.stdout:148: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-27T09:35:06.875 INFO:teuthology.orchestra.run.smithi170.stderr:reweighted osd.2 to 1 (10000) 2024-03-27T09:35:06.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-27T09:35:07.011 INFO:tasks.rados.rados.0.smithi170.stdout:146: finishing write tid 1 to smithi17031214-31 2024-03-27T09:35:07.011 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 3 to smithi17031214-14 2024-03-27T09:35:07.011 INFO:tasks.rados.rados.0.smithi170.stdout:142: finishing write tid 1 to smithi17031214-36 2024-03-27T09:35:07.011 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 4 to smithi17031214-14 2024-03-27T09:35:07.011 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 5 to smithi17031214-14 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 14 updating version 0 to 20 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 14 version 20 is already newer than 19 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 14 updating version 20 to 22 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 14 version 22 is already newer than 22 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 22 (ObjNum 93 snap 7 seq_num 93) dirty exists 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: left oid 14 (ObjNum 93 snap 7 seq_num 93) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:146: finishing write tid 2 to smithi17031214-31 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:146: finishing write tid 3 to smithi17031214-31 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing write tid 1 to smithi17031214-18 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:142: finishing write tid 2 to smithi17031214-36 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:141: done (11 left) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:143: done (10 left) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:144: done (9 left) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:145: done (8 left) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:147: done (7 left) 2024-03-27T09:35:07.012 INFO:tasks.rados.rados.0.smithi170.stdout:148: done (6 left) 2024-03-27T09:35:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:154: done (5 left) 2024-03-27T09:35:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:155: read oid 15 snap -1 2024-03-27T09:35:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:155: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-27T09:35:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:156: write (excl) oid 27 current snap is 8 2024-03-27T09:35:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:156: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-03-27T09:35:07.017 INFO:tasks.rados.rados.0.smithi170.stdout:156: writing smithi17031214-27 from 463893 to 1026133 tid 1 2024-03-27T09:35:07.023 INFO:tasks.rados.rados.0.smithi170.stdout:156: writing smithi17031214-27 from 1593482 to 2251509 tid 2 2024-03-27T09:35:07.030 INFO:tasks.rados.rados.0.smithi170.stdout:156: writing smithi17031214-27 from 2685935 to 3245276 tid 3 2024-03-27T09:35:07.030 INFO:tasks.rados.rados.0.smithi170.stdout:157: delete oid 6 current snap is 8 2024-03-27T09:35:07.030 INFO:tasks.rados.rados.0.smithi170.stdout:146: finishing write tid 4 to smithi17031214-31 2024-03-27T09:35:07.030 INFO:tasks.rados.rados.0.smithi170.stdout:146: finishing write tid 5 to smithi17031214-31 2024-03-27T09:35:07.030 INFO:tasks.rados.rados.0.smithi170.stdout:146: oid 31 updating version 0 to 27 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:146: oid 31 version 27 is already newer than 25 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:146: oid 31 updating version 27 to 28 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:146: oid 31 version 28 is already newer than 27 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 28 (ObjNum 94 snap 7 seq_num 94) dirty exists 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:146: left oid 31 (ObjNum 94 snap 7 seq_num 94) 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing write tid 2 to smithi17031214-18 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing write tid 3 to smithi17031214-18 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:142: finishing write tid 3 to smithi17031214-36 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing write tid 4 to smithi17031214-18 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing write tid 5 to smithi17031214-18 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: oid 18 updating version 0 to 39 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: oid 18 updating version 39 to 40 2024-03-27T09:35:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:150: oid 18 version 40 is already newer than 37 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:150: oid 18 version 40 is already newer than 38 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 40 (ObjNum 96 snap 7 seq_num 96) dirty exists 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:150: left oid 18 (ObjNum 96 snap 7 seq_num 96) 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: finishing write tid 4 to smithi17031214-36 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: finishing write tid 5 to smithi17031214-36 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: oid 36 updating version 0 to 49 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: oid 36 version 49 is already newer than 47 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: oid 36 updating version 49 to 50 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: oid 36 version 50 is already newer than 49 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 50 (ObjNum 90 snap 7 seq_num 90) dirty exists 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:142: left oid 36 (ObjNum 90 snap 7 seq_num 90) 2024-03-27T09:35:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:153: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:07.058+0000 7f2793ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:07.072 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:07.072+0000 7f2793ffa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:07.146 INFO:tasks.rados.rados.0.smithi170.stdout:152: finishing write tid 1 to smithi17031214-40 2024-03-27T09:35:07.146 INFO:tasks.rados.rados.0.smithi170.stdout:152: finishing write tid 2 to smithi17031214-40 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: finishing write tid 3 to smithi17031214-40 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: finishing write tid 4 to smithi17031214-40 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: finishing write tid 5 to smithi17031214-40 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: oid 40 updating version 0 to 54 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: oid 40 version 54 is already newer than 52 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: oid 40 version 54 is already newer than 51 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: oid 40 version 54 is already newer than 54 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 54 (ObjNum 98 snap 7 seq_num 98) dirty exists 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:152: left oid 40 (ObjNum 98 snap 7 seq_num 98) 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:142: done (7 left) 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:146: done (6 left) 2024-03-27T09:35:07.147 INFO:tasks.rados.rados.0.smithi170.stdout:150: done (5 left) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:152: done (4 left) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:153: done (3 left) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:157: done (2 left) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:158: read oid 45 snap 7 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:158: expect (ObjNum 70 snap 3 seq_num 70) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:159: read oid 5 snap -1 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:159: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:160: snap_create 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing write tid 1 to smithi17031214-27 2024-03-27T09:35:07.148 INFO:tasks.rados.rados.0.smithi170.stdout:155: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing write tid 2 to smithi17031214-27 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing write tid 3 to smithi17031214-27 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing write tid 4 to smithi17031214-27 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing write tid 5 to smithi17031214-27 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: oid 27 updating version 0 to 57 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: oid 27 updating version 57 to 58 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: oid 27 updating version 58 to 60 2024-03-27T09:35:07.240 INFO:tasks.rados.rados.0.smithi170.stdout:156: oid 27 version 60 is already newer than 59 2024-03-27T09:35:07.241 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 60 (ObjNum 99 snap 8 seq_num 99) dirty exists 2024-03-27T09:35:07.241 INFO:tasks.rados.rados.0.smithi170.stdout:156: left oid 27 (ObjNum 99 snap 8 seq_num 99) 2024-03-27T09:35:07.241 INFO:tasks.rados.rados.0.smithi170.stdout:159: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-27T09:35:07.318 INFO:tasks.rados.rados.0.smithi170.stdout:158: expect (ObjNum 70 snap 3 seq_num 70) 2024-03-27T09:35:07.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:07.722 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:07.719+0000 7f27ec6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:07.726 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:07.726+0000 7f27ec6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:07.881 INFO:teuthology.orchestra.run.smithi170.stderr:reweighted osd.3 to 1 (10000) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:155: done (4 left) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:156: done (3 left) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:158: done (2 left) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:159: done (1 left) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:160: done (0 left) 2024-03-27T09:35:07.884 INFO:tasks.rados.rados.0.smithi170.stdout:161: snap_remove snap 8 2024-03-27T09:35:07.895 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-27T09:35:07.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:35:08.005 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.001+0000 7f49b3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.009 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.008+0000 7f49b3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.192 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:35:08.193 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:35:08.205 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-27T09:35:07.882457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117817741,"num_objects":66,"num_object_clones":17,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":355,"num_read_kb":126977,"num_write":453,"num_write_kb":87649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":302360,"kb_used_data":195064,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377185000,"statfs":{"total":386547056640,"available":386237440000,"internally_reserved":0,"allocated":199745536,"data_stored":202935003,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009376"},"pg_stats":[{"pgid":"3.f","version":"23'18","reported_seq":79,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.886279+0000","last_change":"2024-03-27T09:35:02.897975+0000","last_active":"2024-03-27T09:35:02.886279+0000","last_peered":"2024-03-27T09:35:02.886279+0000","last_clean":"2024-03-27T09:35:02.886279+0000","last_became_active":"2024-03-27T09:34:48.951324+0000","last_became_peered":"2024-03-27T09:34:48.951324+0000","last_unstale":"2024-03-27T09:35:02.886279+0000","last_undegraded":"2024-03-27T09:35:02.886279+0000","last_fullsized":"2024-03-27T09:35:02.886279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4689087,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":16096,"num_write":17,"num_write_kb":3913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'31","reported_seq":70,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.246703+0000","last_change":"2024-03-27T09:34:58.024475+0000","last_active":"2024-03-27T09:35:01.246703+0000","last_peered":"2024-03-27T09:35:01.246703+0000","last_clean":"2024-03-27T09:35:01.246703+0000","last_became_active":"2024-03-27T09:34:49.018740+0000","last_became_peered":"2024-03-27T09:34:49.018740+0000","last_unstale":"2024-03-27T09:35:01.246703+0000","last_undegraded":"2024-03-27T09:35:01.246703+0000","last_fullsized":"2024-03-27T09:35:01.246703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8268945,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":5732,"num_write":27,"num_write_kb":6318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"26'27","reported_seq":76,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.295166+0000","last_change":"2024-03-27T09:34:57.986259+0000","last_active":"2024-03-27T09:35:01.295166+0000","last_peered":"2024-03-27T09:35:01.295166+0000","last_clean":"2024-03-27T09:35:01.295166+0000","last_became_active":"2024-03-27T09:34:48.937792+0000","last_became_peered":"2024-03-27T09:34:48.937792+0000","last_unstale":"2024-03-27T09:35:01.295166+0000","last_undegraded":"2024-03-27T09:35:01.295166+0000","last_fullsized":"2024-03-27T09:35:01.295166+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5663893,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":10326,"num_write":29,"num_write_kb":7174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'40","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.894889+0000","last_change":"2024-03-27T09:35:02.898885+0000","last_active":"2024-03-27T09:35:02.894889+0000","last_peered":"2024-03-27T09:35:02.894889+0000","last_clean":"2024-03-27T09:35:02.894889+0000","last_became_active":"2024-03-27T09:34:48.969295+0000","last_became_peered":"2024-03-27T09:34:48.969295+0000","last_unstale":"2024-03-27T09:35:02.894889+0000","last_undegraded":"2024-03-27T09:35:02.894889+0000","last_fullsized":"2024-03-27T09:35:02.894889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10974839,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":5060,"num_write":40,"num_write_kb":11076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'3","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.225601+0000","last_change":"2024-03-27T09:34:57.886974+0000","last_active":"2024-03-27T09:35:01.225601+0000","last_peered":"2024-03-27T09:35:01.225601+0000","last_clean":"2024-03-27T09:35:01.225601+0000","last_became_active":"2024-03-27T09:34:49.019732+0000","last_became_peered":"2024-03-27T09:34:49.019732+0000","last_unstale":"2024-03-27T09:35:01.225601+0000","last_undegraded":"2024-03-27T09:35:01.225601+0000","last_fullsized":"2024-03-27T09:35:01.225601+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"28'30","reported_seq":95,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.890858+0000","last_change":"2024-03-27T09:35:02.914860+0000","last_active":"2024-03-27T09:35:02.890858+0000","last_peered":"2024-03-27T09:35:02.890858+0000","last_clean":"2024-03-27T09:35:02.890858+0000","last_became_active":"2024-03-27T09:34:48.976415+0000","last_became_peered":"2024-03-27T09:34:48.976415+0000","last_unstale":"2024-03-27T09:35:02.890858+0000","last_undegraded":"2024-03-27T09:35:02.890858+0000","last_fullsized":"2024-03-27T09:35:02.890858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16571504,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":15114,"num_write":33,"num_write_kb":10085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"26'20","reported_seq":82,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.893029+0000","last_change":"2024-03-27T09:35:02.900369+0000","last_active":"2024-03-27T09:35:02.893029+0000","last_peered":"2024-03-27T09:35:02.893029+0000","last_clean":"2024-03-27T09:35:02.893029+0000","last_became_active":"2024-03-27T09:34:48.971193+0000","last_became_peered":"2024-03-27T09:34:48.971193+0000","last_unstale":"2024-03-27T09:35:02.893029+0000","last_undegraded":"2024-03-27T09:35:02.893029+0000","last_fullsized":"2024-03-27T09:35:02.893029+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705927,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":12209,"num_write":22,"num_write_kb":6419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'47","reported_seq":94,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.902699+0000","last_change":"2024-03-27T09:34:58.011428+0000","last_active":"2024-03-27T09:35:00.902699+0000","last_peered":"2024-03-27T09:35:00.902699+0000","last_clean":"2024-03-27T09:35:00.902699+0000","last_became_active":"2024-03-27T09:34:49.005787+0000","last_became_peered":"2024-03-27T09:34:49.005787+0000","last_unstale":"2024-03-27T09:35:00.902699+0000","last_undegraded":"2024-03-27T09:35:00.902699+0000","last_fullsized":"2024-03-27T09:35:00.902699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15109328,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":12901,"num_write":47,"num_write_kb":14044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":173,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.194928+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:35:01.194928+0000","last_peered":"2024-03-27T09:35:01.194928+0000","last_clean":"2024-03-27T09:35:01.194928+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:35:01.194928+0000","last_undegraded":"2024-03-27T09:35:01.194928+0000","last_fullsized":"2024-03-27T09:35:01.194928+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.897761+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:35:00.897761+0000","last_peered":"2024-03-27T09:35:00.897761+0000","last_clean":"2024-03-27T09:35:00.897761+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:35:00.897761+0000","last_undegraded":"2024-03-27T09:35:00.897761+0000","last_fullsized":"2024-03-27T09:35:00.897761+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.896691+0000","last_change":"2024-03-27T09:34:57.912909+0000","last_active":"2024-03-27T09:35:00.896691+0000","last_peered":"2024-03-27T09:35:00.896691+0000","last_clean":"2024-03-27T09:35:00.896691+0000","last_became_active":"2024-03-27T09:34:48.987725+0000","last_became_peered":"2024-03-27T09:34:48.987725+0000","last_unstale":"2024-03-27T09:35:00.896691+0000","last_undegraded":"2024-03-27T09:35:00.896691+0000","last_fullsized":"2024-03-27T09:35:00.896691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.887315+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:35:02.887315+0000","last_peered":"2024-03-27T09:35:02.887315+0000","last_clean":"2024-03-27T09:35:02.887315+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:35:02.887315+0000","last_undegraded":"2024-03-27T09:35:02.887315+0000","last_fullsized":"2024-03-27T09:35:02.887315+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'3","reported_seq":45,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.227087+0000","last_change":"2024-03-27T09:34:57.885938+0000","last_active":"2024-03-27T09:35:01.227087+0000","last_peered":"2024-03-27T09:35:01.227087+0000","last_clean":"2024-03-27T09:35:01.227087+0000","last_became_active":"2024-03-27T09:34:49.047522+0000","last_became_peered":"2024-03-27T09:34:49.047522+0000","last_unstale":"2024-03-27T09:35:01.227087+0000","last_undegraded":"2024-03-27T09:35:01.227087+0000","last_fullsized":"2024-03-27T09:35:01.227087+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4007,"num_write":4,"num_write_kb":995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.881650+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:35:00.881650+0000","last_peered":"2024-03-27T09:35:00.881650+0000","last_clean":"2024-03-27T09:35:00.881650+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:35:00.881650+0000","last_undegraded":"2024-03-27T09:35:00.881650+0000","last_fullsized":"2024-03-27T09:35:00.881650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'14","reported_seq":55,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.208190+0000","last_change":"2024-03-27T09:34:57.969328+0000","last_active":"2024-03-27T09:35:01.208190+0000","last_peered":"2024-03-27T09:35:01.208190+0000","last_clean":"2024-03-27T09:35:01.208190+0000","last_became_active":"2024-03-27T09:34:48.981602+0000","last_became_peered":"2024-03-27T09:34:48.981602+0000","last_unstale":"2024-03-27T09:35:01.208190+0000","last_undegraded":"2024-03-27T09:35:01.208190+0000","last_fullsized":"2024-03-27T09:35:01.208190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378174,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7270,"num_write":12,"num_write_kb":3147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":55,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.888145+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:35:02.888145+0000","last_peered":"2024-03-27T09:35:02.888145+0000","last_clean":"2024-03-27T09:35:02.888145+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:35:02.888145+0000","last_undegraded":"2024-03-27T09:35:02.888145+0000","last_fullsized":"2024-03-27T09:35:02.888145+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'9","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.879800+0000","last_change":"2024-03-27T09:34:57.907728+0000","last_active":"2024-03-27T09:35:00.879800+0000","last_peered":"2024-03-27T09:35:00.879800+0000","last_clean":"2024-03-27T09:35:00.879800+0000","last_became_active":"2024-03-27T09:34:48.945926+0000","last_became_peered":"2024-03-27T09:34:48.945926+0000","last_unstale":"2024-03-27T09:35:00.879800+0000","last_undegraded":"2024-03-27T09:35:00.879800+0000","last_fullsized":"2024-03-27T09:35:00.879800+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479016,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'20","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.886658+0000","last_change":"2024-03-27T09:34:57.898383+0000","last_active":"2024-03-27T09:35:00.886658+0000","last_peered":"2024-03-27T09:35:00.886658+0000","last_clean":"2024-03-27T09:35:00.886658+0000","last_became_active":"2024-03-27T09:34:48.958003+0000","last_became_peered":"2024-03-27T09:34:48.958003+0000","last_unstale":"2024-03-27T09:35:00.886658+0000","last_undegraded":"2024-03-27T09:35:00.886658+0000","last_fullsized":"2024-03-27T09:35:00.886658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9882384,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":11213,"num_write":26,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.893705+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:35:00.893705+0000","last_peered":"2024-03-27T09:35:00.893705+0000","last_clean":"2024-03-27T09:35:00.893705+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:35:00.893705+0000","last_undegraded":"2024-03-27T09:35:00.893705+0000","last_fullsized":"2024-03-27T09:35:00.893705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'6","reported_seq":49,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.891804+0000","last_change":"2024-03-27T09:34:57.910961+0000","last_active":"2024-03-27T09:35:00.891804+0000","last_peered":"2024-03-27T09:35:00.891804+0000","last_clean":"2024-03-27T09:35:00.891804+0000","last_became_active":"2024-03-27T09:34:48.995086+0000","last_became_peered":"2024-03-27T09:34:48.995086+0000","last_unstale":"2024-03-27T09:35:00.891804+0000","last_undegraded":"2024-03-27T09:35:00.891804+0000","last_fullsized":"2024-03-27T09:35:00.891804+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4621859,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4368,"num_write":8,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.223850+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:35:01.223850+0000","last_peered":"2024-03-27T09:35:01.223850+0000","last_clean":"2024-03-27T09:35:01.223850+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:35:01.223850+0000","last_undegraded":"2024-03-27T09:35:01.223850+0000","last_fullsized":"2024-03-27T09:35:01.223850+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"26'34","reported_seq":86,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.400919+0000","last_change":"2024-03-27T09:34:57.887861+0000","last_active":"2024-03-27T09:35:01.400919+0000","last_peered":"2024-03-27T09:35:01.400919+0000","last_clean":"2024-03-27T09:35:01.400919+0000","last_became_active":"2024-03-27T09:34:49.013275+0000","last_became_peered":"2024-03-27T09:34:49.013275+0000","last_unstale":"2024-03-27T09:35:01.400919+0000","last_undegraded":"2024-03-27T09:35:01.400919+0000","last_fullsized":"2024-03-27T09:35:01.400919+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13680733,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":22586,"num_write":35,"num_write_kb":10098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.897117+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:35:00.897117+0000","last_peered":"2024-03-27T09:35:00.897117+0000","last_clean":"2024-03-27T09:35:00.897117+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:35:00.897117+0000","last_undegraded":"2024-03-27T09:35:00.897117+0000","last_fullsized":"2024-03-27T09:35:00.897117+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.881781+0000","last_change":"2024-03-27T09:34:57.906199+0000","last_active":"2024-03-27T09:35:00.881781+0000","last_peered":"2024-03-27T09:35:00.881781+0000","last_clean":"2024-03-27T09:35:00.881781+0000","last_became_active":"2024-03-27T09:34:48.967426+0000","last_became_peered":"2024-03-27T09:34:48.967426+0000","last_unstale":"2024-03-27T09:35:00.881781+0000","last_undegraded":"2024-03-27T09:35:00.881781+0000","last_fullsized":"2024-03-27T09:35:00.881781+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.899875+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:35:00.899875+0000","last_peered":"2024-03-27T09:35:00.899875+0000","last_clean":"2024-03-27T09:35:00.899875+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:35:00.899875+0000","last_undegraded":"2024-03-27T09:35:00.899875+0000","last_fullsized":"2024-03-27T09:35:00.899875+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":117227373,"num_objects":64,"num_object_clones":17,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":252,"num_read_kb":126886,"num_write":316,"num_write_kb":85755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77376,"kb_used_data":50552,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294464,"statfs":{"total":96636764160,"available":96557531136,"internally_reserved":0,"allocated":51765248,"data_stored":52731466,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75104,"kb_used_data":48280,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296736,"statfs":{"total":96636764160,"available":96559857664,"internally_reserved":0,"allocated":49438720,"data_stored":49667620,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78056,"kb_used_data":51232,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293784,"statfs":{"total":96636764160,"available":96556834816,"internally_reserved":0,"allocated":52461568,"data_stored":53163072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71824,"kb_used_data":45000,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300016,"statfs":{"total":96636764160,"available":96563216384,"internally_reserved":0,"allocated":46080000,"data_stored":47372845,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:35:08.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:35:08.316 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.312+0000 7f2bdb46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.319+0000 7f2bdb46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.503 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:35:08.503 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:35:08.515 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-27T09:35:07.882457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117817741,"num_objects":66,"num_object_clones":17,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":355,"num_read_kb":126977,"num_write":453,"num_write_kb":87649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":302360,"kb_used_data":195064,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377185000,"statfs":{"total":386547056640,"available":386237440000,"internally_reserved":0,"allocated":199745536,"data_stored":202935003,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009376"},"pg_stats":[{"pgid":"3.f","version":"23'18","reported_seq":79,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.886279+0000","last_change":"2024-03-27T09:35:02.897975+0000","last_active":"2024-03-27T09:35:02.886279+0000","last_peered":"2024-03-27T09:35:02.886279+0000","last_clean":"2024-03-27T09:35:02.886279+0000","last_became_active":"2024-03-27T09:34:48.951324+0000","last_became_peered":"2024-03-27T09:34:48.951324+0000","last_unstale":"2024-03-27T09:35:02.886279+0000","last_undegraded":"2024-03-27T09:35:02.886279+0000","last_fullsized":"2024-03-27T09:35:02.886279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4689087,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":16096,"num_write":17,"num_write_kb":3913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"23'31","reported_seq":70,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.246703+0000","last_change":"2024-03-27T09:34:58.024475+0000","last_active":"2024-03-27T09:35:01.246703+0000","last_peered":"2024-03-27T09:35:01.246703+0000","last_clean":"2024-03-27T09:35:01.246703+0000","last_became_active":"2024-03-27T09:34:49.018740+0000","last_became_peered":"2024-03-27T09:34:49.018740+0000","last_unstale":"2024-03-27T09:35:01.246703+0000","last_undegraded":"2024-03-27T09:35:01.246703+0000","last_fullsized":"2024-03-27T09:35:01.246703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8268945,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":5732,"num_write":27,"num_write_kb":6318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"26'27","reported_seq":76,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.295166+0000","last_change":"2024-03-27T09:34:57.986259+0000","last_active":"2024-03-27T09:35:01.295166+0000","last_peered":"2024-03-27T09:35:01.295166+0000","last_clean":"2024-03-27T09:35:01.295166+0000","last_became_active":"2024-03-27T09:34:48.937792+0000","last_became_peered":"2024-03-27T09:34:48.937792+0000","last_unstale":"2024-03-27T09:35:01.295166+0000","last_undegraded":"2024-03-27T09:35:01.295166+0000","last_fullsized":"2024-03-27T09:35:01.295166+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5663893,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":10326,"num_write":29,"num_write_kb":7174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"24'40","reported_seq":96,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.894889+0000","last_change":"2024-03-27T09:35:02.898885+0000","last_active":"2024-03-27T09:35:02.894889+0000","last_peered":"2024-03-27T09:35:02.894889+0000","last_clean":"2024-03-27T09:35:02.894889+0000","last_became_active":"2024-03-27T09:34:48.969295+0000","last_became_peered":"2024-03-27T09:34:48.969295+0000","last_unstale":"2024-03-27T09:35:02.894889+0000","last_undegraded":"2024-03-27T09:35:02.894889+0000","last_fullsized":"2024-03-27T09:35:02.894889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10974839,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":5060,"num_write":40,"num_write_kb":11076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'3","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.225601+0000","last_change":"2024-03-27T09:34:57.886974+0000","last_active":"2024-03-27T09:35:01.225601+0000","last_peered":"2024-03-27T09:35:01.225601+0000","last_clean":"2024-03-27T09:35:01.225601+0000","last_became_active":"2024-03-27T09:34:49.019732+0000","last_became_peered":"2024-03-27T09:34:49.019732+0000","last_unstale":"2024-03-27T09:35:01.225601+0000","last_undegraded":"2024-03-27T09:35:01.225601+0000","last_fullsized":"2024-03-27T09:35:01.225601+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"28'30","reported_seq":95,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.890858+0000","last_change":"2024-03-27T09:35:02.914860+0000","last_active":"2024-03-27T09:35:02.890858+0000","last_peered":"2024-03-27T09:35:02.890858+0000","last_clean":"2024-03-27T09:35:02.890858+0000","last_became_active":"2024-03-27T09:34:48.976415+0000","last_became_peered":"2024-03-27T09:34:48.976415+0000","last_unstale":"2024-03-27T09:35:02.890858+0000","last_undegraded":"2024-03-27T09:35:02.890858+0000","last_fullsized":"2024-03-27T09:35:02.890858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16571504,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":15114,"num_write":33,"num_write_kb":10085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"26'20","reported_seq":82,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.893029+0000","last_change":"2024-03-27T09:35:02.900369+0000","last_active":"2024-03-27T09:35:02.893029+0000","last_peered":"2024-03-27T09:35:02.893029+0000","last_clean":"2024-03-27T09:35:02.893029+0000","last_became_active":"2024-03-27T09:34:48.971193+0000","last_became_peered":"2024-03-27T09:34:48.971193+0000","last_unstale":"2024-03-27T09:35:02.893029+0000","last_undegraded":"2024-03-27T09:35:02.893029+0000","last_fullsized":"2024-03-27T09:35:02.893029+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705927,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":23,"num_read_kb":12209,"num_write":22,"num_write_kb":6419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"23'47","reported_seq":94,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.902699+0000","last_change":"2024-03-27T09:34:58.011428+0000","last_active":"2024-03-27T09:35:00.902699+0000","last_peered":"2024-03-27T09:35:00.902699+0000","last_clean":"2024-03-27T09:35:00.902699+0000","last_became_active":"2024-03-27T09:34:49.005787+0000","last_became_peered":"2024-03-27T09:34:49.005787+0000","last_unstale":"2024-03-27T09:35:00.902699+0000","last_undegraded":"2024-03-27T09:35:00.902699+0000","last_fullsized":"2024-03-27T09:35:00.902699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15109328,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":12901,"num_write":47,"num_write_kb":14044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":173,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.194928+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:35:01.194928+0000","last_peered":"2024-03-27T09:35:01.194928+0000","last_clean":"2024-03-27T09:35:01.194928+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:35:01.194928+0000","last_undegraded":"2024-03-27T09:35:01.194928+0000","last_fullsized":"2024-03-27T09:35:01.194928+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.897761+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:35:00.897761+0000","last_peered":"2024-03-27T09:35:00.897761+0000","last_clean":"2024-03-27T09:35:00.897761+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:35:00.897761+0000","last_undegraded":"2024-03-27T09:35:00.897761+0000","last_fullsized":"2024-03-27T09:35:00.897761+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.896691+0000","last_change":"2024-03-27T09:34:57.912909+0000","last_active":"2024-03-27T09:35:00.896691+0000","last_peered":"2024-03-27T09:35:00.896691+0000","last_clean":"2024-03-27T09:35:00.896691+0000","last_became_active":"2024-03-27T09:34:48.987725+0000","last_became_peered":"2024-03-27T09:34:48.987725+0000","last_unstale":"2024-03-27T09:35:00.896691+0000","last_undegraded":"2024-03-27T09:35:00.896691+0000","last_fullsized":"2024-03-27T09:35:00.896691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.887315+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:35:02.887315+0000","last_peered":"2024-03-27T09:35:02.887315+0000","last_clean":"2024-03-27T09:35:02.887315+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:35:02.887315+0000","last_undegraded":"2024-03-27T09:35:02.887315+0000","last_fullsized":"2024-03-27T09:35:02.887315+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'3","reported_seq":45,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.227087+0000","last_change":"2024-03-27T09:34:57.885938+0000","last_active":"2024-03-27T09:35:01.227087+0000","last_peered":"2024-03-27T09:35:01.227087+0000","last_clean":"2024-03-27T09:35:01.227087+0000","last_became_active":"2024-03-27T09:34:49.047522+0000","last_became_peered":"2024-03-27T09:34:49.047522+0000","last_unstale":"2024-03-27T09:35:01.227087+0000","last_undegraded":"2024-03-27T09:35:01.227087+0000","last_fullsized":"2024-03-27T09:35:01.227087+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2050421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4007,"num_write":4,"num_write_kb":995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.881650+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:35:00.881650+0000","last_peered":"2024-03-27T09:35:00.881650+0000","last_clean":"2024-03-27T09:35:00.881650+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:35:00.881650+0000","last_undegraded":"2024-03-27T09:35:00.881650+0000","last_fullsized":"2024-03-27T09:35:00.881650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'14","reported_seq":55,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.208190+0000","last_change":"2024-03-27T09:34:57.969328+0000","last_active":"2024-03-27T09:35:01.208190+0000","last_peered":"2024-03-27T09:35:01.208190+0000","last_clean":"2024-03-27T09:35:01.208190+0000","last_became_active":"2024-03-27T09:34:48.981602+0000","last_became_peered":"2024-03-27T09:34:48.981602+0000","last_unstale":"2024-03-27T09:35:01.208190+0000","last_undegraded":"2024-03-27T09:35:01.208190+0000","last_fullsized":"2024-03-27T09:35:01.208190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378174,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7270,"num_write":12,"num_write_kb":3147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":55,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-27T09:35:02.888145+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:35:02.888145+0000","last_peered":"2024-03-27T09:35:02.888145+0000","last_clean":"2024-03-27T09:35:02.888145+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:35:02.888145+0000","last_undegraded":"2024-03-27T09:35:02.888145+0000","last_fullsized":"2024-03-27T09:35:02.888145+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'9","reported_seq":51,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.879800+0000","last_change":"2024-03-27T09:34:57.907728+0000","last_active":"2024-03-27T09:35:00.879800+0000","last_peered":"2024-03-27T09:35:00.879800+0000","last_clean":"2024-03-27T09:35:00.879800+0000","last_became_active":"2024-03-27T09:34:48.945926+0000","last_became_peered":"2024-03-27T09:34:48.945926+0000","last_unstale":"2024-03-27T09:35:00.879800+0000","last_undegraded":"2024-03-27T09:35:00.879800+0000","last_fullsized":"2024-03-27T09:35:00.879800+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479016,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"23'20","reported_seq":73,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.886658+0000","last_change":"2024-03-27T09:34:57.898383+0000","last_active":"2024-03-27T09:35:00.886658+0000","last_peered":"2024-03-27T09:35:00.886658+0000","last_clean":"2024-03-27T09:35:00.886658+0000","last_became_active":"2024-03-27T09:34:48.958003+0000","last_became_peered":"2024-03-27T09:34:48.958003+0000","last_unstale":"2024-03-27T09:35:00.886658+0000","last_undegraded":"2024-03-27T09:35:00.886658+0000","last_fullsized":"2024-03-27T09:35:00.886658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9882384,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":11213,"num_write":26,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.893705+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:35:00.893705+0000","last_peered":"2024-03-27T09:35:00.893705+0000","last_clean":"2024-03-27T09:35:00.893705+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:35:00.893705+0000","last_undegraded":"2024-03-27T09:35:00.893705+0000","last_fullsized":"2024-03-27T09:35:00.893705+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'6","reported_seq":49,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.891804+0000","last_change":"2024-03-27T09:34:57.910961+0000","last_active":"2024-03-27T09:35:00.891804+0000","last_peered":"2024-03-27T09:35:00.891804+0000","last_clean":"2024-03-27T09:35:00.891804+0000","last_became_active":"2024-03-27T09:34:48.995086+0000","last_became_peered":"2024-03-27T09:34:48.995086+0000","last_unstale":"2024-03-27T09:35:00.891804+0000","last_undegraded":"2024-03-27T09:35:00.891804+0000","last_fullsized":"2024-03-27T09:35:00.891804+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4621859,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4368,"num_write":8,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.223850+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:35:01.223850+0000","last_peered":"2024-03-27T09:35:01.223850+0000","last_clean":"2024-03-27T09:35:01.223850+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:35:01.223850+0000","last_undegraded":"2024-03-27T09:35:01.223850+0000","last_fullsized":"2024-03-27T09:35:01.223850+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"26'34","reported_seq":86,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:01.400919+0000","last_change":"2024-03-27T09:34:57.887861+0000","last_active":"2024-03-27T09:35:01.400919+0000","last_peered":"2024-03-27T09:35:01.400919+0000","last_clean":"2024-03-27T09:35:01.400919+0000","last_became_active":"2024-03-27T09:34:49.013275+0000","last_became_peered":"2024-03-27T09:34:49.013275+0000","last_unstale":"2024-03-27T09:35:01.400919+0000","last_undegraded":"2024-03-27T09:35:01.400919+0000","last_fullsized":"2024-03-27T09:35:01.400919+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13680733,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":22586,"num_write":35,"num_write_kb":10098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.897117+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:35:00.897117+0000","last_peered":"2024-03-27T09:35:00.897117+0000","last_clean":"2024-03-27T09:35:00.897117+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:35:00.897117+0000","last_undegraded":"2024-03-27T09:35:00.897117+0000","last_fullsized":"2024-03-27T09:35:00.897117+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.881781+0000","last_change":"2024-03-27T09:34:57.906199+0000","last_active":"2024-03-27T09:35:00.881781+0000","last_peered":"2024-03-27T09:35:00.881781+0000","last_clean":"2024-03-27T09:35:00.881781+0000","last_became_active":"2024-03-27T09:34:48.967426+0000","last_became_peered":"2024-03-27T09:34:48.967426+0000","last_unstale":"2024-03-27T09:35:00.881781+0000","last_undegraded":"2024-03-27T09:35:00.881781+0000","last_fullsized":"2024-03-27T09:35:00.881781+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-27T09:35:00.899875+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:35:00.899875+0000","last_peered":"2024-03-27T09:35:00.899875+0000","last_clean":"2024-03-27T09:35:00.899875+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:35:00.899875+0000","last_undegraded":"2024-03-27T09:35:00.899875+0000","last_fullsized":"2024-03-27T09:35:00.899875+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":117227373,"num_objects":64,"num_object_clones":17,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":252,"num_read_kb":126886,"num_write":316,"num_write_kb":85755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77376,"kb_used_data":50552,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294464,"statfs":{"total":96636764160,"available":96557531136,"internally_reserved":0,"allocated":51765248,"data_stored":52731466,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75104,"kb_used_data":48280,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94296736,"statfs":{"total":96636764160,"available":96559857664,"internally_reserved":0,"allocated":49438720,"data_stored":49667620,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78056,"kb_used_data":51232,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293784,"statfs":{"total":96636764160,"available":96556834816,"internally_reserved":0,"allocated":52461568,"data_stored":53163072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71824,"kb_used_data":45000,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300016,"statfs":{"total":96636764160,"available":96563216384,"internally_reserved":0,"allocated":46080000,"data_stored":47372845,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:35:08.516 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-27T09:35:08.516 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:08.517 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-27T09:35:08.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-27T09:35:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.623+0000 7f503366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.632 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.631+0000 7f503366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:08.648 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:08.651 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:08.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-27T09:35:08.763 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.758+0000 7fbdd4aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.767 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.766+0000 7fbdd4aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:08.783 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:08.784 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:08.784 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:08.784 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:08.784 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:08.787 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:08.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-27T09:35:08.885 INFO:tasks.rados.rados.0.smithi170.stdout:161: done (0 left) 2024-03-27T09:35:08.885 INFO:tasks.rados.rados.0.smithi170.stdout:162: write (excl) oid 26 current snap is 9 2024-03-27T09:35:08.885 INFO:tasks.rados.rados.0.smithi170.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-03-27T09:35:08.891 INFO:tasks.rados.rados.0.smithi170.stdout:162: writing smithi17031214-26 from 497507 to 1210245 tid 1 2024-03-27T09:35:08.900 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.894+0000 7f5ea7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.900 INFO:tasks.rados.rados.0.smithi170.stdout:162: writing smithi17031214-26 from 1710446 to 2372142 tid 2 2024-03-27T09:35:08.905 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:08.905+0000 7f5ea7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:08.910 INFO:tasks.rados.rados.0.smithi170.stdout:162: writing smithi17031214-26 from 3140781 to 3697507 tid 3 2024-03-27T09:35:08.910 INFO:tasks.rados.rados.0.smithi170.stdout:163: snap_remove snap 7 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:08.938 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:08.939 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:08.939 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:08.939 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:08.942 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:08.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-27T09:35:09.071 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:09.065+0000 7f59bfdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:09.075 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:09.074+0000 7f59bfdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:09.091 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:09.092 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:09.094 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:09.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:09.892 INFO:tasks.rados.rados.0.smithi170.stdout:163: done (1 left) 2024-03-27T09:35:09.892 INFO:tasks.rados.rados.0.smithi170.stdout:164: read oid 7 snap -1 2024-03-27T09:35:09.892 INFO:tasks.rados.rados.0.smithi170.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:165: delete oid 12 current snap is 9 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing write tid 1 to smithi17031214-26 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing write tid 2 to smithi17031214-26 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing write tid 3 to smithi17031214-26 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing write tid 4 to smithi17031214-26 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing write tid 5 to smithi17031214-26 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: oid 26 updating version 0 to 35 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: oid 26 version 35 is already newer than 34 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: oid 26 version 35 is already newer than 33 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: oid 26 version 35 is already newer than 32 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 35 (ObjNum 101 snap 9 seq_num 101) dirty exists 2024-03-27T09:35:09.893 INFO:tasks.rados.rados.0.smithi170.stdout:162: left oid 26 (ObjNum 101 snap 9 seq_num 101) 2024-03-27T09:35:09.956 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:09.954+0000 7f8b6e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:09.960 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:09.960+0000 7f8b6e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:09.992 INFO:tasks.rados.rados.0.smithi170.stdout:162: done (2 left) 2024-03-27T09:35:09.993 INFO:tasks.rados.rados.0.smithi170.stdout:165: done (1 left) 2024-03-27T09:35:09.993 INFO:tasks.rados.rados.0.smithi170.stdout:166: delete oid 10 current snap is 9 2024-03-27T09:35:10.004 INFO:tasks.rados.rados.0.smithi170.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-27T09:35:10.168 INFO:tasks.rados.rados.0.smithi170.stdout:164: done (1 left) 2024-03-27T09:35:10.168 INFO:tasks.rados.rados.0.smithi170.stdout:166: done (0 left) 2024-03-27T09:35:10.169 INFO:tasks.rados.rados.0.smithi170.stdout:167: delete oid 11 current snap is 9 2024-03-27T09:35:10.186 INFO:tasks.rados.rados.0.smithi170.stdout:167: done (0 left) 2024-03-27T09:35:10.187 INFO:tasks.rados.rados.0.smithi170.stdout:168: write (excl) oid 44 current snap is 9 2024-03-27T09:35:10.187 INFO:tasks.rados.rados.0.smithi170.stdout:168: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-03-27T09:35:10.193 INFO:tasks.rados.rados.0.smithi170.stdout:168: writing smithi17031214-44 from 564735 to 1178469 tid 1 2024-03-27T09:35:10.200 INFO:tasks.rados.rados.0.smithi170.stdout:168: writing smithi17031214-44 from 1828021 to 2497055 tid 2 2024-03-27T09:35:10.207 INFO:tasks.rados.rados.0.smithi170.stdout:168: writing smithi17031214-44 from 3134120 to 3764735 tid 3 2024-03-27T09:35:10.207 INFO:tasks.rados.rados.0.smithi170.stdout:169: write oid 2 current snap is 9 2024-03-27T09:35:10.207 INFO:tasks.rados.rados.0.smithi170.stdout:169: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-03-27T09:35:10.214 INFO:tasks.rados.rados.0.smithi170.stdout:169: writing smithi17031214-2 from 581542 to 1270525 tid 1 2024-03-27T09:35:10.221 INFO:tasks.rados.rados.0.smithi170.stdout:169: writing smithi17031214-2 from 1886503 to 2299195 tid 2 2024-03-27T09:35:10.228 INFO:tasks.rados.rados.0.smithi170.stdout:169: writing smithi17031214-2 from 2761543 to 3432494 tid 3 2024-03-27T09:35:10.228 INFO:tasks.rados.rados.0.smithi170.stdout:168: finishing write tid 1 to smithi17031214-44 2024-03-27T09:35:10.229 INFO:tasks.rados.rados.0.smithi170.stdout:170: read oid 21 snap -1 2024-03-27T09:35:10.229 INFO:tasks.rados.rados.0.smithi170.stdout:170: expect deleted 2024-03-27T09:35:10.229 INFO:tasks.rados.rados.0.smithi170.stdout:171: write oid 17 current snap is 9 2024-03-27T09:35:10.229 INFO:tasks.rados.rados.0.smithi170.stdout:171: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-03-27T09:35:10.234 INFO:tasks.rados.rados.0.smithi170.stdout:171: writing smithi17031214-17 from 598349 to 1078934 tid 1 2024-03-27T09:35:10.242 INFO:tasks.rados.rados.0.smithi170.stdout:171: writing smithi17031214-17 from 1544985 to 2217688 tid 2 2024-03-27T09:35:10.249 INFO:tasks.rados.rados.0.smithi170.stdout:171: writing smithi17031214-17 from 2905319 to 3387542 tid 3 2024-03-27T09:35:10.249 INFO:tasks.rados.rados.0.smithi170.stdout:172: write (excl) oid 15 current snap is 9 2024-03-27T09:35:10.249 INFO:tasks.rados.rados.0.smithi170.stdout:172: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-03-27T09:35:10.256 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17031214-15 from 615156 to 1170990 tid 1 2024-03-27T09:35:10.262 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17031214-15 from 1603467 to 2019828 tid 2 2024-03-27T09:35:10.271 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17031214-15 from 2816389 to 3509884 tid 3 2024-03-27T09:35:10.271 INFO:tasks.rados.rados.0.smithi170.stdout:173: read oid 9 snap -1 2024-03-27T09:35:10.271 INFO:tasks.rados.rados.0.smithi170.stdout:173: expect (ObjNum 91 snap 7 seq_num 91) 2024-03-27T09:35:10.271 INFO:tasks.rados.rados.0.smithi170.stdout:174: write (excl) oid 39 current snap is 9 2024-03-27T09:35:10.272 INFO:tasks.rados.rados.0.smithi170.stdout:174: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-03-27T09:35:10.278 INFO:tasks.rados.rados.0.smithi170.stdout:174: writing smithi17031214-39 from 631963 to 1263046 tid 1 2024-03-27T09:35:10.286 INFO:tasks.rados.rados.0.smithi170.stdout:174: writing smithi17031214-39 from 2061949 to 2621968 tid 2 2024-03-27T09:35:10.295 INFO:tasks.rados.rados.0.smithi170.stdout:174: writing smithi17031214-39 from 3243812 to 3748579 tid 3 2024-03-27T09:35:10.295 INFO:tasks.rados.rados.0.smithi170.stdout:175: read oid 3 snap -1 2024-03-27T09:35:10.295 INFO:tasks.rados.rados.0.smithi170.stdout:175: expect (ObjNum 87 snap 6 seq_num 87) 2024-03-27T09:35:10.295 INFO:tasks.rados.rados.0.smithi170.stdout:176: write oid 35 current snap is 9 2024-03-27T09:35:10.295 INFO:tasks.rados.rados.0.smithi170.stdout:176: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-03-27T09:35:10.306 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17031214-35 from 648770 to 1355102 tid 1 2024-03-27T09:35:10.314 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17031214-35 from 2120431 to 2540461 tid 2 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17031214-35 from 3271235 to 3703627 tid 3 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:177: snap_create 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:168: finishing write tid 2 to smithi17031214-44 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:169: finishing write tid 1 to smithi17031214-2 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:168: finishing write tid 3 to smithi17031214-44 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 1 to smithi17031214-15 2024-03-27T09:35:10.321 INFO:tasks.rados.rados.0.smithi170.stdout:169: finishing write tid 2 to smithi17031214-2 2024-03-27T09:35:10.324 INFO:tasks.rados.rados.0.smithi170.stdout:168: finishing write tid 4 to smithi17031214-44 2024-03-27T09:35:10.330 INFO:tasks.rados.rados.0.smithi170.stdout:174: finishing write tid 1 to smithi17031214-39 2024-03-27T09:35:10.335 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 2 to smithi17031214-15 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:169: finishing write tid 3 to smithi17031214-2 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: finishing write tid 5 to smithi17031214-44 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: oid 44 updating version 0 to 51 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: oid 44 updating version 51 to 52 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: oid 44 updating version 52 to 54 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: oid 44 version 54 is already newer than 54 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 54 (ObjNum 105 snap 9 seq_num 105) dirty exists 2024-03-27T09:35:10.340 INFO:tasks.rados.rados.0.smithi170.stdout:168: left oid 44 (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:35:10.371 INFO:tasks.rados.rados.0.smithi170.stdout:169: finishing write tid 4 to smithi17031214-2 2024-03-27T09:35:10.373 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing write tid 1 to smithi17031214-17 2024-03-27T09:35:10.373 INFO:tasks.rados.rados.0.smithi170.stdout:169: finishing write tid 5 to smithi17031214-2 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:169: oid 2 updating version 0 to 65 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:169: oid 2 updating version 65 to 66 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:169: oid 2 version 66 is already newer than 64 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:169: oid 2 version 66 is already newer than 63 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 66 (ObjNum 106 snap 9 seq_num 106) dirty exists 2024-03-27T09:35:10.374 INFO:tasks.rados.rados.0.smithi170.stdout:169: left oid 2 (ObjNum 106 snap 9 seq_num 106) 2024-03-27T09:35:10.385 INFO:tasks.rados.rados.0.smithi170.stdout:174: finishing write tid 2 to smithi17031214-39 2024-03-27T09:35:10.391 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 3 to smithi17031214-15 2024-03-27T09:35:10.406 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 4 to smithi17031214-15 2024-03-27T09:35:10.407 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 5 to smithi17031214-15 2024-03-27T09:35:10.407 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 15 updating version 0 to 7 2024-03-27T09:35:10.407 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 15 updating version 7 to 10 2024-03-27T09:35:10.407 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 15 version 10 is already newer than 10 2024-03-27T09:35:10.408 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 15 version 10 is already newer than 8 2024-03-27T09:35:10.408 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 10 (ObjNum 108 snap 9 seq_num 108) dirty exists 2024-03-27T09:35:10.408 INFO:tasks.rados.rados.0.smithi170.stdout:172: left oid 15 (ObjNum 108 snap 9 seq_num 108) 2024-03-27T09:35:10.416 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing write tid 2 to smithi17031214-17 2024-03-27T09:35:10.438 INFO:tasks.rados.rados.0.smithi170.stdout:174: finishing write tid 3 to smithi17031214-39 2024-03-27T09:35:10.439 INFO:tasks.rados.rados.0.smithi170.stdout:173: expect (ObjNum 91 snap 7 seq_num 91) 2024-03-27T09:35:10.620 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 1 to smithi17031214-35 2024-03-27T09:35:10.621 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing write tid 3 to smithi17031214-17 2024-03-27T09:35:10.621 INFO:tasks.rados.rados.0.smithi170.stdout:175: expect (ObjNum 87 snap 6 seq_num 87) 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: finishing write tid 4 to smithi17031214-39 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: finishing write tid 5 to smithi17031214-39 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: oid 39 updating version 0 to 10 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: oid 39 updating version 10 to 11 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: oid 39 updating version 11 to 13 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: oid 39 version 13 is already newer than 12 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 13 (ObjNum 109 snap 9 seq_num 109) dirty exists 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:174: left oid 39 (ObjNum 109 snap 9 seq_num 109) 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing write tid 4 to smithi17031214-17 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing write tid 5 to smithi17031214-17 2024-03-27T09:35:10.783 INFO:tasks.rados.rados.0.smithi170.stdout:171: oid 17 updating version 0 to 56 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:171: oid 17 version 56 is already newer than 55 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:171: oid 17 updating version 56 to 57 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:171: oid 17 updating version 57 to 58 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 58 (ObjNum 107 snap 9 seq_num 107) dirty exists 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:171: left oid 17 (ObjNum 107 snap 9 seq_num 107) 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 2 to smithi17031214-35 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 3 to smithi17031214-35 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 4 to smithi17031214-35 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 5 to smithi17031214-35 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 35 updating version 0 to 43 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 35 version 43 is already newer than 43 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 35 updating version 43 to 44 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 35 version 44 is already newer than 41 2024-03-27T09:35:10.784 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 44 (ObjNum 110 snap 9 seq_num 110) dirty exists 2024-03-27T09:35:10.785 INFO:tasks.rados.rados.0.smithi170.stdout:176: left oid 35 (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:35:10.893 INFO:tasks.rados.rados.0.smithi170.stdout:168: done (9 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:169: done (8 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:170: done (7 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:171: done (6 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:172: done (5 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:173: done (4 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:174: done (3 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:175: done (2 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:176: done (1 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:177: done (0 left) 2024-03-27T09:35:10.894 INFO:tasks.rados.rados.0.smithi170.stdout:178: delete oid 23 current snap is 10 2024-03-27T09:35:10.928 INFO:tasks.rados.rados.0.smithi170.stdout:178: done (0 left) 2024-03-27T09:35:10.928 INFO:tasks.rados.rados.0.smithi170.stdout:179: delete oid 36 current snap is 10 2024-03-27T09:35:10.964 INFO:tasks.rados.rados.0.smithi170.stdout:179: done (0 left) 2024-03-27T09:35:10.964 INFO:tasks.rados.rados.0.smithi170.stdout:180: write oid 41 current snap is 10 2024-03-27T09:35:10.964 INFO:tasks.rados.rados.0.smithi170.stdout:180: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-03-27T09:35:10.970 INFO:tasks.rados.rados.0.smithi170.stdout:180: writing smithi17031214-41 from 699191 to 1231270 tid 1 2024-03-27T09:35:10.978 INFO:tasks.rados.rados.0.smithi170.stdout:180: writing smithi17031214-41 from 1779524 to 2346881 tid 2 2024-03-27T09:35:10.986 INFO:tasks.rados.rados.0.smithi170.stdout:180: writing smithi17031214-41 from 2837151 to 3503359 tid 3 2024-03-27T09:35:10.987 INFO:tasks.rados.rados.0.smithi170.stdout:181: write oid 20 current snap is 10 2024-03-27T09:35:10.987 INFO:tasks.rados.rados.0.smithi170.stdout:181: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-03-27T09:35:10.995 INFO:tasks.rados.rados.0.smithi170.stdout:181: writing smithi17031214-20 from 715998 to 1323326 tid 1 2024-03-27T09:35:11.001 INFO:tasks.rados.rados.0.smithi170.stdout:181: writing smithi17031214-20 from 1838006 to 2265374 tid 2 2024-03-27T09:35:11.007 INFO:tasks.rados.rados.0.smithi170.stdout:181: writing smithi17031214-20 from 2864574 to 3342054 tid 3 2024-03-27T09:35:11.011 INFO:tasks.rados.rados.0.smithi170.stdout:181: writing smithi17031214-20 from 3863835 to 3915998 tid 4 2024-03-27T09:35:11.011 INFO:tasks.rados.rados.0.smithi170.stdout:182: snap_remove snap 9 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:182: done (2 left) 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:183: delete oid 30 current snap is 10 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: finishing write tid 1 to smithi17031214-41 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: finishing write tid 2 to smithi17031214-41 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: finishing write tid 3 to smithi17031214-41 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 1 to smithi17031214-20 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: finishing write tid 4 to smithi17031214-41 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: finishing write tid 5 to smithi17031214-41 2024-03-27T09:35:11.898 INFO:tasks.rados.rados.0.smithi170.stdout:180: oid 41 updating version 0 to 46 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:180: oid 41 updating version 46 to 48 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:180: oid 41 version 48 is already newer than 48 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:180: oid 41 updating version 48 to 49 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 49 (ObjNum 113 snap 10 seq_num 113) dirty exists 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:180: left oid 41 (ObjNum 113 snap 10 seq_num 113) 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 2 to smithi17031214-20 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 3 to smithi17031214-20 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 4 to smithi17031214-20 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 5 to smithi17031214-20 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: finishing write tid 6 to smithi17031214-20 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: oid 20 updating version 0 to 35 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: oid 20 updating version 35 to 36 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: oid 20 version 36 is already newer than 34 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: oid 20 version 36 is already newer than 33 2024-03-27T09:35:11.899 INFO:tasks.rados.rados.0.smithi170.stdout:181: oid 20 version 36 is already newer than 32 2024-03-27T09:35:11.900 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 36 (ObjNum 114 snap 10 seq_num 114) dirty exists 2024-03-27T09:35:11.900 INFO:tasks.rados.rados.0.smithi170.stdout:181: left oid 20 (ObjNum 114 snap 10 seq_num 114) 2024-03-27T09:35:11.963 INFO:tasks.rados.rados.0.smithi170.stdout:180: done (2 left) 2024-03-27T09:35:11.963 INFO:tasks.rados.rados.0.smithi170.stdout:181: done (1 left) 2024-03-27T09:35:11.963 INFO:tasks.rados.rados.0.smithi170.stdout:183: done (0 left) 2024-03-27T09:35:11.963 INFO:tasks.rados.rados.0.smithi170.stdout:184: write (excl) oid 45 current snap is 10 2024-03-27T09:35:11.963 INFO:tasks.rados.rados.0.smithi170.stdout:184: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-03-27T09:35:11.970 INFO:tasks.rados.rados.0.smithi170.stdout:184: writing smithi17031214-45 from 749612 to 1223791 tid 1 2024-03-27T09:35:11.977 INFO:tasks.rados.rados.0.smithi170.stdout:184: writing smithi17031214-45 from 1954970 to 2386007 tid 2 2024-03-27T09:35:11.984 INFO:tasks.rados.rados.0.smithi170.stdout:184: writing smithi17031214-45 from 2919420 to 3535797 tid 3 2024-03-27T09:35:11.984 INFO:tasks.rados.rados.0.smithi170.stdout:185: write (excl) oid 4 current snap is 10 2024-03-27T09:35:11.984 INFO:tasks.rados.rados.0.smithi170.stdout:185: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-03-27T09:35:11.992 INFO:tasks.rados.rados.0.smithi170.stdout:185: writing smithi17031214-4 from 766419 to 1315847 tid 1 2024-03-27T09:35:12.000 INFO:tasks.rados.rados.0.smithi170.stdout:185: writing smithi17031214-4 from 2013452 to 2588147 tid 2 2024-03-27T09:35:12.008 INFO:tasks.rados.rados.0.smithi170.stdout:185: writing smithi17031214-4 from 3346843 to 3774492 tid 3 2024-03-27T09:35:12.008 INFO:tasks.rados.rados.0.smithi170.stdout:186: delete oid 30 current snap is 10 2024-03-27T09:35:12.024 INFO:tasks.rados.rados.0.smithi170.stdout:186: done (2 left) 2024-03-27T09:35:12.024 INFO:tasks.rados.rados.0.smithi170.stdout:187: write (excl) oid 3 current snap is 10 2024-03-27T09:35:12.024 INFO:tasks.rados.rados.0.smithi170.stdout:187: seq_num 119 ranges {400033=699926,1614063=385970} 2024-03-27T09:35:12.031 INFO:tasks.rados.rados.0.smithi170.stdout:187: writing smithi17031214-3 from 400033 to 1099959 tid 1 2024-03-27T09:35:12.037 INFO:tasks.rados.rados.0.smithi170.stdout:187: writing smithi17031214-3 from 1614063 to 2000033 tid 2 2024-03-27T09:35:12.037 INFO:tasks.rados.rados.0.smithi170.stdout:188: write oid 26 current snap is 10 2024-03-27T09:35:12.037 INFO:tasks.rados.rados.0.smithi170.stdout:188: seq_num 120 ranges {416840=775175,1672545=344295} 2024-03-27T09:35:12.046 INFO:tasks.rados.rados.0.smithi170.stdout:188: writing smithi17031214-26 from 416840 to 1192015 tid 1 2024-03-27T09:35:12.053 INFO:tasks.rados.rados.0.smithi170.stdout:188: writing smithi17031214-26 from 1672545 to 2016840 tid 2 2024-03-27T09:35:12.053 INFO:tasks.rados.rados.0.smithi170.stdout:189: snap_create 2024-03-27T09:35:12.053 INFO:tasks.rados.rados.0.smithi170.stdout:185: finishing write tid 1 to smithi17031214-4 2024-03-27T09:35:12.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:12.090 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing write tid 1 to smithi17031214-45 2024-03-27T09:35:12.092 INFO:tasks.rados.rados.0.smithi170.stdout:187: finishing write tid 1 to smithi17031214-3 2024-03-27T09:35:12.104 INFO:tasks.rados.rados.0.smithi170.stdout:185: finishing write tid 2 to smithi17031214-4 2024-03-27T09:35:12.131 INFO:tasks.rados.rados.0.smithi170.stdout:188: finishing write tid 1 to smithi17031214-26 2024-03-27T09:35:12.140 INFO:tasks.rados.rados.0.smithi170.stdout:187: finishing write tid 2 to smithi17031214-3 2024-03-27T09:35:12.143 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing write tid 2 to smithi17031214-45 2024-03-27T09:35:12.155 INFO:tasks.rados.rados.0.smithi170.stdout:185: finishing write tid 3 to smithi17031214-4 2024-03-27T09:35:12.169 INFO:tasks.rados.rados.0.smithi170.stdout:187: finishing write tid 3 to smithi17031214-3 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:187: finishing write tid 4 to smithi17031214-3 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:187: oid 3 updating version 0 to 28 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:187: oid 3 version 28 is already newer than 28 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:187: oid 3 version 28 is already newer than 26 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 28 (ObjNum 119 snap 10 seq_num 119) dirty exists 2024-03-27T09:35:12.172 INFO:tasks.rados.rados.0.smithi170.stdout:187: left oid 3 (ObjNum 119 snap 10 seq_num 119) 2024-03-27T09:35:12.175 INFO:tasks.rados.rados.0.smithi170.stdout:188: finishing write tid 2 to smithi17031214-26 2024-03-27T09:35:12.176 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing write tid 3 to smithi17031214-45 2024-03-27T09:35:12.176 INFO:tasks.rados.rados.0.smithi170.stdout:185: finishing write tid 4 to smithi17031214-4 2024-03-27T09:35:12.177 INFO:tasks.rados.rados.0.smithi170.stdout:185: finishing write tid 5 to smithi17031214-4 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:185: oid 4 updating version 0 to 7 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:185: oid 4 version 7 is already newer than 6 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:185: oid 4 version 7 is already newer than 6 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:185: oid 4 version 7 is already newer than 4 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 7 (ObjNum 117 snap 10 seq_num 117) dirty exists 2024-03-27T09:35:12.178 INFO:tasks.rados.rados.0.smithi170.stdout:185: left oid 4 (ObjNum 117 snap 10 seq_num 117) 2024-03-27T09:35:12.195 INFO:tasks.rados.rados.0.smithi170.stdout:188: finishing write tid 3 to smithi17031214-26 2024-03-27T09:35:12.195 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing write tid 4 to smithi17031214-45 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing write tid 5 to smithi17031214-45 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: oid 45 updating version 0 to 70 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: oid 45 version 70 is already newer than 69 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: oid 45 version 70 is already newer than 68 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: oid 45 version 70 is already newer than 67 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 70 (ObjNum 116 snap 10 seq_num 116) dirty exists 2024-03-27T09:35:12.197 INFO:tasks.rados.rados.0.smithi170.stdout:184: left oid 45 (ObjNum 116 snap 10 seq_num 116) 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:188: finishing write tid 4 to smithi17031214-26 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:188: oid 26 updating version 0 to 40 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:188: oid 26 version 40 is already newer than 38 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:188: oid 26 version 40 is already newer than 39 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 40 (ObjNum 120 snap 10 seq_num 120) dirty exists 2024-03-27T09:35:12.198 INFO:tasks.rados.rados.0.smithi170.stdout:188: left oid 26 (ObjNum 120 snap 10 seq_num 120) 2024-03-27T09:35:12.204 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:12.202+0000 7f6a44e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:12.207 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:12.207+0000 7f6a44e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:12.900 INFO:tasks.rados.rados.0.smithi170.stdout:184: done (4 left) 2024-03-27T09:35:12.901 INFO:tasks.rados.rados.0.smithi170.stdout:185: done (3 left) 2024-03-27T09:35:12.901 INFO:tasks.rados.rados.0.smithi170.stdout:187: done (2 left) 2024-03-27T09:35:12.901 INFO:tasks.rados.rados.0.smithi170.stdout:188: done (1 left) 2024-03-27T09:35:12.901 INFO:tasks.rados.rados.0.smithi170.stdout:189: done (0 left) 2024-03-27T09:35:12.901 INFO:tasks.rados.rados.0.smithi170.stdout:190: snap_create 2024-03-27T09:35:13.905 INFO:tasks.rados.rados.0.smithi170.stdout:190: done (0 left) 2024-03-27T09:35:13.905 INFO:tasks.rados.rados.0.smithi170.stdout:191: read oid 2 snap -1 2024-03-27T09:35:13.905 INFO:tasks.rados.rados.0.smithi170.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-03-27T09:35:13.905 INFO:tasks.rados.rados.0.smithi170.stdout:192: snap_create 2024-03-27T09:35:13.966 INFO:tasks.rados.rados.0.smithi170.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-03-27T09:35:14.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:14.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:14.095 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-27T09:35:14.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-27T09:35:14.208 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.202+0000 7f5f3bead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.212 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.211+0000 7f5f3bead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.229 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:14.229 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:14.229 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:14.229 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:14.230 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:14.232 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:14.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-27T09:35:14.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:14.345 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.340+0000 7f8efe35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.349 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.349+0000 7f8efe35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:14.366 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:14.369 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:14.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-27T09:35:14.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.418+0000 7ff65f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.424 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.424+0000 7ff65f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.529 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.523+0000 7fda56006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.532+0000 7fda56006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:14.666 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:14.667 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:14.667 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:14.667 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-27T09:35:14.819 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.812+0000 7fcfd4a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.824 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:14.824+0000 7fcfd4a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:14.876 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-03-27T09:35:14.876 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:config help 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:config show 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:deep_scrub 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_ops 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:dump_historic_slow_ops 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:dump_metrics [] 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:dump_ops_in_flight 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:dump_pgstate_history 2024-03-27T09:35:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:Error EINVAL: invalid command 2024-03-27T09:35:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:14.880 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-27T09:35:14.905 INFO:tasks.rados.rados.0.smithi170.stdout:191: done (1 left) 2024-03-27T09:35:14.905 INFO:tasks.rados.rados.0.smithi170.stdout:192: done (0 left) 2024-03-27T09:35:14.905 INFO:tasks.rados.rados.0.smithi170.stdout:193: snap_create 2024-03-27T09:35:15.910 INFO:tasks.rados.rados.0.smithi170.stdout:193: done (0 left) 2024-03-27T09:35:15.910 INFO:tasks.rados.rados.0.smithi170.stdout:194: delete oid 46 current snap is 14 2024-03-27T09:35:15.978 INFO:tasks.rados.rados.0.smithi170.stdout:194: done (0 left) 2024-03-27T09:35:15.978 INFO:tasks.rados.rados.0.smithi170.stdout:195: snap_remove snap 13 2024-03-27T09:35:16.665 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:16.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:16.781+0000 7f7dd775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:16.787 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:16.786+0000 7f7dd775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:16.934 INFO:tasks.rados.rados.0.smithi170.stdout:195: done (0 left) 2024-03-27T09:35:16.934 INFO:tasks.rados.rados.0.smithi170.stdout:196: write (excl) oid 40 current snap is 14 2024-03-27T09:35:16.934 INFO:tasks.rados.rados.0.smithi170.stdout:196: seq_num 122 ranges {450454=642026,1505862=544592} 2024-03-27T09:35:16.940 INFO:tasks.rados.rados.0.smithi170.stdout:196: writing smithi17031214-40 from 450454 to 1092480 tid 1 2024-03-27T09:35:16.947 INFO:tasks.rados.rados.0.smithi170.stdout:196: writing smithi17031214-40 from 1505862 to 2050454 tid 2 2024-03-27T09:35:16.947 INFO:tasks.rados.rados.0.smithi170.stdout:197: read oid 8 snap 12 2024-03-27T09:35:16.947 INFO:tasks.rados.rados.0.smithi170.stdout:197: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:35:16.947 INFO:tasks.rados.rados.0.smithi170.stdout:198: delete oid 17 current snap is 14 2024-03-27T09:35:17.038 INFO:tasks.rados.rados.0.smithi170.stdout:197: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: finishing write tid 1 to smithi17031214-40 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: finishing write tid 2 to smithi17031214-40 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: finishing write tid 3 to smithi17031214-40 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: finishing write tid 4 to smithi17031214-40 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: oid 40 updating version 0 to 60 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: oid 40 updating version 60 to 61 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: oid 40 version 61 is already newer than 59 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 61 (ObjNum 122 snap 14 seq_num 122) dirty exists 2024-03-27T09:35:17.124 INFO:tasks.rados.rados.0.smithi170.stdout:196: left oid 40 (ObjNum 122 snap 14 seq_num 122) 2024-03-27T09:35:17.125 INFO:tasks.rados.rados.0.smithi170.stdout:196: done (2 left) 2024-03-27T09:35:17.125 INFO:tasks.rados.rados.0.smithi170.stdout:197: done (1 left) 2024-03-27T09:35:17.125 INFO:tasks.rados.rados.0.smithi170.stdout:198: done (0 left) 2024-03-27T09:35:17.125 INFO:tasks.rados.rados.0.smithi170.stdout:199: snap_create 2024-03-27T09:35:17.939 INFO:tasks.rados.rados.0.smithi170.stdout:199: done (0 left) 2024-03-27T09:35:17.939 INFO:tasks.rados.rados.0.smithi170.stdout:200: write (excl) oid 33 current snap is 15 2024-03-27T09:35:17.939 INFO:tasks.rados.rados.0.smithi170.stdout:200: seq_num 124 ranges {484068=792524,1906473=177595} 2024-03-27T09:35:17.947 INFO:tasks.rados.rados.0.smithi170.stdout:200: writing smithi17031214-33 from 484068 to 1276592 tid 1 2024-03-27T09:35:17.953 INFO:tasks.rados.rados.0.smithi170.stdout:200: writing smithi17031214-33 from 1906473 to 2084068 tid 2 2024-03-27T09:35:17.953 INFO:tasks.rados.rados.0.smithi170.stdout:201: read oid 46 snap 12 2024-03-27T09:35:17.953 INFO:tasks.rados.rados.0.smithi170.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-27T09:35:17.953 INFO:tasks.rados.rados.0.smithi170.stdout:202: delete oid 19 current snap is 15 2024-03-27T09:35:17.975 INFO:tasks.rados.rados.0.smithi170.stdout:202: done (2 left) 2024-03-27T09:35:17.975 INFO:tasks.rados.rados.0.smithi170.stdout:203: snap_remove snap 14 2024-03-27T09:35:18.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:18.940 INFO:tasks.rados.rados.0.smithi170.stdout:203: done (2 left) 2024-03-27T09:35:18.940 INFO:tasks.rados.rados.0.smithi170.stdout:204: read oid 32 snap -1 2024-03-27T09:35:18.941 INFO:tasks.rados.rados.0.smithi170.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:18.941 INFO:tasks.rados.rados.0.smithi170.stdout:205: write (excl) oid 5 current snap is 15 2024-03-27T09:35:18.941 INFO:tasks.rados.rados.0.smithi170.stdout:205: seq_num 126 ranges {517682=543022,1623437=494245} 2024-03-27T09:35:18.946 INFO:tasks.rados.rados.0.smithi170.stdout:205: writing smithi17031214-5 from 517682 to 1060704 tid 1 2024-03-27T09:35:18.953 INFO:tasks.rados.rados.0.smithi170.stdout:205: writing smithi17031214-5 from 1623437 to 2117682 tid 2 2024-03-27T09:35:18.953 INFO:tasks.rados.rados.0.smithi170.stdout:206: read oid 31 snap -1 2024-03-27T09:35:18.953 INFO:tasks.rados.rados.0.smithi170.stdout:206: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-27T09:35:18.954 INFO:tasks.rados.rados.0.smithi170.stdout:207: snap_remove snap 10 2024-03-27T09:35:19.029 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:19.027+0000 7fcb4ea24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:19.035 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:19.034+0000 7fcb4ea24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:19.880 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:19.881 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:19.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:35:19.945 INFO:tasks.rados.rados.0.smithi170.stdout:207: done (5 left) 2024-03-27T09:35:19.945 INFO:tasks.rados.rados.0.smithi170.stdout:208: delete oid 8 current snap is 15 2024-03-27T09:35:19.945 INFO:tasks.rados.rados.0.smithi170.stdout:201: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-27T09:35:20.017 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:20.013+0000 7fbf0126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:20.021 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:20.021+0000 7fbf0126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: finishing write tid 1 to smithi17031214-33 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: finishing write tid 2 to smithi17031214-33 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: finishing write tid 3 to smithi17031214-33 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: finishing write tid 4 to smithi17031214-33 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: oid 33 updating version 0 to 32 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: oid 33 version 32 is already newer than 31 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: oid 33 version 32 is already newer than 30 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 32 (ObjNum 124 snap 15 seq_num 124) dirty exists 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: left oid 33 (ObjNum 124 snap 15 seq_num 124) 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:200: done (5 left) 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:201: done (4 left) 2024-03-27T09:35:20.114 INFO:tasks.rados.rados.0.smithi170.stdout:208: done (3 left) 2024-03-27T09:35:20.115 INFO:tasks.rados.rados.0.smithi170.stdout:209: write (excl) oid 15 current snap is 15 2024-03-27T09:35:20.115 INFO:tasks.rados.rados.0.smithi170.stdout:204: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:20.210 INFO:tasks.rados.rados.0.smithi170.stdout:206: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-27T09:35:20.214 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:35:20.214 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:35:20.226 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-03-27T09:35:19.941916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112809618,"num_objects":73,"num_object_clones":27,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":480,"num_read_kb":222455,"num_write":610,"num_write_kb":133380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":560,"ondisk_log_size":560,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":310796,"kb_used_data":203500,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377176564,"statfs":{"total":386547056640,"available":386228801536,"internally_reserved":0,"allocated":208384000,"data_stored":208251418,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8890771,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":19,"num_read_kb":19568,"num_write":10,"num_write_kb":2107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033118"},"pg_stats":[{"pgid":"3.f","version":"37'24","reported_seq":133,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.979401+0000","last_change":"2024-03-27T09:35:16.993860+0000","last_active":"2024-03-27T09:35:17.979401+0000","last_peered":"2024-03-27T09:35:17.979401+0000","last_clean":"2024-03-27T09:35:17.979401+0000","last_became_active":"2024-03-27T09:34:48.951324+0000","last_became_peered":"2024-03-27T09:34:48.951324+0000","last_unstale":"2024-03-27T09:35:17.979401+0000","last_undegraded":"2024-03-27T09:35:17.979401+0000","last_fullsized":"2024-03-27T09:35:17.979401+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764735,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":19472,"num_write":23,"num_write_kb":5444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"35'35","reported_seq":119,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.958628+0000","last_change":"2024-03-27T09:35:16.965118+0000","last_active":"2024-03-27T09:35:16.958628+0000","last_peered":"2024-03-27T09:35:16.958628+0000","last_clean":"2024-03-27T09:35:16.958628+0000","last_became_active":"2024-03-27T09:34:49.018740+0000","last_became_peered":"2024-03-27T09:34:49.018740+0000","last_unstale":"2024-03-27T09:35:16.958628+0000","last_undegraded":"2024-03-27T09:35:16.958628+0000","last_fullsized":"2024-03-27T09:35:16.958628+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159683,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":10737,"num_write":28,"num_write_kb":6318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"37'40","reported_seq":135,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.952515+0000","last_change":"2024-03-27T09:35:16.991174+0000","last_active":"2024-03-27T09:35:17.952515+0000","last_peered":"2024-03-27T09:35:17.952515+0000","last_clean":"2024-03-27T09:35:17.952515+0000","last_became_active":"2024-03-27T09:34:48.937792+0000","last_became_peered":"2024-03-27T09:34:48.937792+0000","last_unstale":"2024-03-27T09:35:17.952515+0000","last_undegraded":"2024-03-27T09:35:17.952515+0000","last_fullsized":"2024-03-27T09:35:17.952515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016840,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":10328,"num_write":39,"num_write_kb":10157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"42'63","reported_seq":170,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:18.011395+0000","last_change":"2024-03-27T09:35:17.109889+0000","last_active":"2024-03-27T09:35:18.011395+0000","last_peered":"2024-03-27T09:35:18.011395+0000","last_clean":"2024-03-27T09:35:18.011395+0000","last_became_active":"2024-03-27T09:34:48.969295+0000","last_became_peered":"2024-03-27T09:34:48.969295+0000","last_unstale":"2024-03-27T09:35:18.011395+0000","last_undegraded":"2024-03-27T09:35:18.011395+0000","last_fullsized":"2024-03-27T09:35:18.011395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11613538,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":49,"num_read_kb":28039,"num_write":60,"num_write_kb":16920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"35'4","reported_seq":88,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.995354+0000","last_change":"2024-03-27T09:35:17.008063+0000","last_active":"2024-03-27T09:35:16.995354+0000","last_peered":"2024-03-27T09:35:16.995354+0000","last_clean":"2024-03-27T09:35:16.995354+0000","last_became_active":"2024-03-27T09:34:49.019732+0000","last_became_peered":"2024-03-27T09:34:49.019732+0000","last_unstale":"2024-03-27T09:35:16.995354+0000","last_undegraded":"2024-03-27T09:35:16.995354+0000","last_fullsized":"2024-03-27T09:35:16.995354+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"37'51","reported_seq":163,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.980012+0000","last_change":"2024-03-27T09:35:16.979126+0000","last_active":"2024-03-27T09:35:17.980012+0000","last_peered":"2024-03-27T09:35:17.980012+0000","last_clean":"2024-03-27T09:35:17.980012+0000","last_became_active":"2024-03-27T09:34:48.976415+0000","last_became_peered":"2024-03-27T09:34:48.976415+0000","last_unstale":"2024-03-27T09:35:17.980012+0000","last_undegraded":"2024-03-27T09:35:17.980012+0000","last_fullsized":"2024-03-27T09:35:17.980012+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19798448,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":20082,"num_write":53,"num_write_kb":16285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"43'32","reported_seq":147,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:18.036112+0000","last_change":"2024-03-27T09:35:16.978430+0000","last_active":"2024-03-27T09:35:18.036112+0000","last_peered":"2024-03-27T09:35:18.036112+0000","last_clean":"2024-03-27T09:35:18.036112+0000","last_became_active":"2024-03-27T09:34:48.971193+0000","last_became_peered":"2024-03-27T09:34:48.971193+0000","last_unstale":"2024-03-27T09:35:18.036112+0000","last_undegraded":"2024-03-27T09:35:18.036112+0000","last_fullsized":"2024-03-27T09:35:18.036112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428661,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":25837,"num_write":36,"num_write_kb":10238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"37'70","reported_seq":177,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.961867+0000","last_change":"2024-03-27T09:35:17.007959+0000","last_active":"2024-03-27T09:35:17.961867+0000","last_peered":"2024-03-27T09:35:17.961867+0000","last_clean":"2024-03-27T09:35:17.961867+0000","last_became_active":"2024-03-27T09:34:49.005787+0000","last_became_peered":"2024-03-27T09:34:49.005787+0000","last_unstale":"2024-03-27T09:35:17.961867+0000","last_undegraded":"2024-03-27T09:35:17.961867+0000","last_fullsized":"2024-03-27T09:35:17.961867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17042133,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":38070,"num_write":72,"num_write_kb":22135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":209,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.982164+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:35:16.982164+0000","last_peered":"2024-03-27T09:35:16.982164+0000","last_clean":"2024-03-27T09:35:16.982164+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:35:16.982164+0000","last_undegraded":"2024-03-27T09:35:16.982164+0000","last_fullsized":"2024-03-27T09:35:16.982164+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":88,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.964849+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:35:17.964849+0000","last_peered":"2024-03-27T09:35:17.964849+0000","last_clean":"2024-03-27T09:35:17.964849+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:35:17.964849+0000","last_undegraded":"2024-03-27T09:35:17.964849+0000","last_fullsized":"2024-03-27T09:35:17.964849+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":90,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.960495+0000","last_change":"2024-03-27T09:35:17.007234+0000","last_active":"2024-03-27T09:35:17.960495+0000","last_peered":"2024-03-27T09:35:17.960495+0000","last_clean":"2024-03-27T09:35:17.960495+0000","last_became_active":"2024-03-27T09:34:48.987725+0000","last_became_peered":"2024-03-27T09:34:48.987725+0000","last_unstale":"2024-03-27T09:35:17.960495+0000","last_undegraded":"2024-03-27T09:35:17.960495+0000","last_fullsized":"2024-03-27T09:35:17.960495+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":94,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.981719+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:35:17.981719+0000","last_peered":"2024-03-27T09:35:17.981719+0000","last_clean":"2024-03-27T09:35:17.981719+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:35:17.981719+0000","last_undegraded":"2024-03-27T09:35:17.981719+0000","last_fullsized":"2024-03-27T09:35:17.981719+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'7","reported_seq":94,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.996434+0000","last_change":"2024-03-27T09:35:17.008363+0000","last_active":"2024-03-27T09:35:16.996434+0000","last_peered":"2024-03-27T09:35:16.996434+0000","last_clean":"2024-03-27T09:35:16.996434+0000","last_became_active":"2024-03-27T09:34:49.047522+0000","last_became_peered":"2024-03-27T09:34:49.047522+0000","last_unstale":"2024-03-27T09:35:16.996434+0000","last_undegraded":"2024-03-27T09:35:16.996434+0000","last_fullsized":"2024-03-27T09:35:16.996434+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3966419,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4008,"num_write":9,"num_write_kb":2512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.950826+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:35:17.950826+0000","last_peered":"2024-03-27T09:35:17.950826+0000","last_clean":"2024-03-27T09:35:17.950826+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:35:17.950826+0000","last_undegraded":"2024-03-27T09:35:17.950826+0000","last_fullsized":"2024-03-27T09:35:17.950826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"23'14","reported_seq":105,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.995933+0000","last_change":"2024-03-27T09:35:17.000170+0000","last_active":"2024-03-27T09:35:16.995933+0000","last_peered":"2024-03-27T09:35:16.995933+0000","last_clean":"2024-03-27T09:35:16.995933+0000","last_became_active":"2024-03-27T09:34:48.981602+0000","last_became_peered":"2024-03-27T09:34:48.981602+0000","last_unstale":"2024-03-27T09:35:16.995933+0000","last_undegraded":"2024-03-27T09:35:16.995933+0000","last_fullsized":"2024-03-27T09:35:16.995933+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378174,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":19819,"num_write":12,"num_write_kb":3147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":94,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.981883+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:35:17.981883+0000","last_peered":"2024-03-27T09:35:17.981883+0000","last_clean":"2024-03-27T09:35:17.981883+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:35:17.981883+0000","last_undegraded":"2024-03-27T09:35:17.981883+0000","last_fullsized":"2024-03-27T09:35:17.981883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"35'13","reported_seq":104,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.943318+0000","last_change":"2024-03-27T09:35:16.958644+0000","last_active":"2024-03-27T09:35:16.943318+0000","last_peered":"2024-03-27T09:35:16.943318+0000","last_clean":"2024-03-27T09:35:16.943318+0000","last_became_active":"2024-03-27T09:34:48.945926+0000","last_became_peered":"2024-03-27T09:34:48.945926+0000","last_unstale":"2024-03-27T09:35:16.943318+0000","last_undegraded":"2024-03-27T09:35:16.943318+0000","last_fullsized":"2024-03-27T09:35:16.943318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672313,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":3050,"num_write":17,"num_write_kb":5004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"37'38","reported_seq":140,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.949707+0000","last_change":"2024-03-27T09:35:16.990885+0000","last_active":"2024-03-27T09:35:17.949707+0000","last_peered":"2024-03-27T09:35:17.949707+0000","last_clean":"2024-03-27T09:35:17.949707+0000","last_became_active":"2024-03-27T09:34:48.958003+0000","last_became_peered":"2024-03-27T09:34:48.958003+0000","last_unstale":"2024-03-27T09:35:17.949707+0000","last_undegraded":"2024-03-27T09:35:17.949707+0000","last_fullsized":"2024-03-27T09:35:17.949707+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966452,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":11216,"num_write":44,"num_write_kb":10935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.954116+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:35:16.954116+0000","last_peered":"2024-03-27T09:35:16.954116+0000","last_clean":"2024-03-27T09:35:16.954116+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:35:16.954116+0000","last_undegraded":"2024-03-27T09:35:16.954116+0000","last_fullsized":"2024-03-27T09:35:16.954116+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"35'10","reported_seq":102,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.960761+0000","last_change":"2024-03-27T09:35:16.965723+0000","last_active":"2024-03-27T09:35:16.960761+0000","last_peered":"2024-03-27T09:35:16.960761+0000","last_clean":"2024-03-27T09:35:16.960761+0000","last_became_active":"2024-03-27T09:34:48.995086+0000","last_became_peered":"2024-03-27T09:34:48.995086+0000","last_unstale":"2024-03-27T09:35:16.960761+0000","last_undegraded":"2024-03-27T09:35:16.960761+0000","last_fullsized":"2024-03-27T09:35:16.960761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6201717,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":6197,"num_write":13,"num_write_kb":3468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.993006+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:35:16.993006+0000","last_peered":"2024-03-27T09:35:16.993006+0000","last_clean":"2024-03-27T09:35:16.993006+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:35:16.993006+0000","last_undegraded":"2024-03-27T09:35:16.993006+0000","last_fullsized":"2024-03-27T09:35:16.993006+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'62","reported_seq":159,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.032441+0000","last_change":"2024-03-27T09:35:17.006683+0000","last_active":"2024-03-27T09:35:17.032441+0000","last_peered":"2024-03-27T09:35:17.032441+0000","last_clean":"2024-03-27T09:35:17.032441+0000","last_became_active":"2024-03-27T09:34:49.013275+0000","last_became_peered":"2024-03-27T09:34:49.013275+0000","last_unstale":"2024-03-27T09:35:17.032441+0000","last_undegraded":"2024-03-27T09:35:17.032441+0000","last_fullsized":"2024-03-27T09:35:17.032441+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210137,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":25508,"num_write":62,"num_write_kb":17896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.956176+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:35:16.956176+0000","last_peered":"2024-03-27T09:35:16.956176+0000","last_clean":"2024-03-27T09:35:16.956176+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:35:16.956176+0000","last_undegraded":"2024-03-27T09:35:16.956176+0000","last_fullsized":"2024-03-27T09:35:16.956176+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-27T09:35:16.952211+0000","last_change":"2024-03-27T09:35:16.959802+0000","last_active":"2024-03-27T09:35:16.952211+0000","last_peered":"2024-03-27T09:35:16.952211+0000","last_clean":"2024-03-27T09:35:16.952211+0000","last_became_active":"2024-03-27T09:34:48.967426+0000","last_became_peered":"2024-03-27T09:34:48.967426+0000","last_unstale":"2024-03-27T09:35:16.952211+0000","last_undegraded":"2024-03-27T09:35:16.952211+0000","last_fullsized":"2024-03-27T09:35:16.952211+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-27T09:35:17.962791+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:35:17.962791+0000","last_peered":"2024-03-27T09:35:17.962791+0000","last_clean":"2024-03-27T09:35:17.962791+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:35:17.962791+0000","last_undegraded":"2024-03-27T09:35:17.962791+0000","last_fullsized":"2024-03-27T09:35:17.962791+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112219250,"num_objects":71,"num_object_clones":27,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":377,"num_read_kb":222364,"num_write":473,"num_write_kb":131486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":463,"ondisk_log_size":463,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79404,"kb_used_data":52580,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292436,"statfs":{"total":96636764160,"available":96555454464,"internally_reserved":0,"allocated":53841920,"data_stored":54013459,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80668,"kb_used_data":53844,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94291172,"statfs":{"total":96636764160,"available":96554160128,"internally_reserved":0,"allocated":55136256,"data_stored":55269393,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77688,"kb_used_data":50864,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294152,"statfs":{"total":96636764160,"available":96557211648,"internally_reserved":0,"allocated":52084736,"data_stored":51428499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":73036,"kb_used_data":46212,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298804,"statfs":{"total":96636764160,"available":96561975296,"internally_reserved":0,"allocated":47321088,"data_stored":47540067,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:35:20.436 INFO:tasks.rados.rados.0.smithi170.stdout:205: finishing write tid 1 to smithi17031214-5 2024-03-27T09:35:20.436 INFO:tasks.rados.rados.0.smithi170.stdout:205: finishing write tid 2 to smithi17031214-5 2024-03-27T09:35:20.436 INFO:tasks.rados.rados.0.smithi170.stdout:209: seq_num 128 ranges {551296=693520,2024048=127248} 2024-03-27T09:35:20.442 INFO:tasks.rados.rados.0.smithi170.stdout:209: writing smithi17031214-15 from 551296 to 1244816 tid 1 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:209: writing smithi17031214-15 from 2024048 to 2151296 tid 2 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:204: done (3 left) 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:206: done (2 left) 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:210: read oid 18 snap -1 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:210: expect (ObjNum 96 snap 7 seq_num 96) 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:211: delete oid 10 current snap is 15 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:205: finishing write tid 3 to smithi17031214-5 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:205: finishing write tid 4 to smithi17031214-5 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:205: oid 5 updating version 0 to 17 2024-03-27T09:35:20.448 INFO:tasks.rados.rados.0.smithi170.stdout:205: oid 5 version 17 is already newer than 16 2024-03-27T09:35:20.449 INFO:tasks.rados.rados.0.smithi170.stdout:205: oid 5 version 17 is already newer than 15 2024-03-27T09:35:20.449 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 17 (ObjNum 126 snap 15 seq_num 126) dirty exists 2024-03-27T09:35:20.449 INFO:tasks.rados.rados.0.smithi170.stdout:205: left oid 5 (ObjNum 126 snap 15 seq_num 126) 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:205: done (3 left) 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:211: done (2 left) 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:212: read oid 16 snap -1 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:212: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:213: write oid 38 current snap is 15 2024-03-27T09:35:20.458 INFO:tasks.rados.rados.0.smithi170.stdout:213: seq_num 130 ranges {584910=560371,1857365=327545} 2024-03-27T09:35:20.464 INFO:tasks.rados.rados.0.smithi170.stdout:213: writing smithi17031214-38 from 584910 to 1145281 tid 1 2024-03-27T09:35:20.472 INFO:tasks.rados.rados.0.smithi170.stdout:213: writing smithi17031214-38 from 1857365 to 2184910 tid 2 2024-03-27T09:35:20.473 INFO:tasks.rados.rados.0.smithi170.stdout:214: write oid 7 current snap is 15 2024-03-27T09:35:20.473 INFO:tasks.rados.rados.0.smithi170.stdout:214: seq_num 131 ranges {601717=635620,1799494=402223} 2024-03-27T09:35:20.482 INFO:tasks.rados.rados.0.smithi170.stdout:214: writing smithi17031214-7 from 601717 to 1237337 tid 1 2024-03-27T09:35:20.489 INFO:tasks.rados.rados.0.smithi170.stdout:214: writing smithi17031214-7 from 1799494 to 2201717 tid 2 2024-03-27T09:35:20.489 INFO:tasks.rados.rados.0.smithi170.stdout:215: snap_remove snap 11 2024-03-27T09:35:20.945 INFO:tasks.rados.rados.0.smithi170.stdout:215: done (5 left) 2024-03-27T09:35:20.946 INFO:tasks.rados.rados.0.smithi170.stdout:216: delete oid 13 current snap is 15 2024-03-27T09:35:20.946 INFO:tasks.rados.rados.0.smithi170.stdout:212: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:21.048 INFO:tasks.rados.rados.0.smithi170.stdout:213: finishing write tid 1 to smithi17031214-38 2024-03-27T09:35:21.048 INFO:tasks.rados.rados.0.smithi170.stdout:210: expect (ObjNum 96 snap 7 seq_num 96) 2024-03-27T09:35:21.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:21.254 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:21.252+0000 7f8bab6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:21.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:21.258+0000 7f8bab6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:214: finishing write tid 1 to smithi17031214-7 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:213: finishing write tid 2 to smithi17031214-38 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:210: done (5 left) 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:212: done (4 left) 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:216: done (3 left) 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:217: write (excl) oid 36 current snap is 15 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:209: finishing write tid 1 to smithi17031214-15 2024-03-27T09:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:217: seq_num 133 ranges {635331=786118,1916458=318873} 2024-03-27T09:35:21.283 INFO:tasks.rados.rados.0.smithi170.stdout:217: writing smithi17031214-36 from 635331 to 1421449 tid 1 2024-03-27T09:35:21.290 INFO:tasks.rados.rados.0.smithi170.stdout:217: writing smithi17031214-36 from 1916458 to 2235331 tid 2 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:218: delete oid 45 current snap is 15 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: finishing write tid 2 to smithi17031214-7 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: finishing write tid 3 to smithi17031214-38 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: finishing write tid 4 to smithi17031214-38 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: oid 38 updating version 0 to 42 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: oid 38 updating version 42 to 44 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: oid 38 version 44 is already newer than 43 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 44 (ObjNum 130 snap 15 seq_num 130) dirty exists 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:213: left oid 38 (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: finishing write tid 3 to smithi17031214-7 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: finishing write tid 4 to smithi17031214-7 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: oid 7 updating version 0 to 17 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: oid 7 version 17 is already newer than 16 2024-03-27T09:35:21.291 INFO:tasks.rados.rados.0.smithi170.stdout:214: oid 7 updating version 17 to 18 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 18 (ObjNum 131 snap 15 seq_num 131) dirty exists 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:214: left oid 7 (ObjNum 131 snap 15 seq_num 131) 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: finishing write tid 2 to smithi17031214-15 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: finishing write tid 3 to smithi17031214-15 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: finishing write tid 4 to smithi17031214-15 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: oid 15 updating version 0 to 13 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: oid 15 updating version 13 to 14 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: oid 15 version 14 is already newer than 12 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 14 (ObjNum 128 snap 15 seq_num 128) dirty exists 2024-03-27T09:35:21.292 INFO:tasks.rados.rados.0.smithi170.stdout:209: left oid 15 (ObjNum 128 snap 15 seq_num 128) 2024-03-27T09:35:21.334 INFO:tasks.rados.rados.0.smithi170.stdout:217: finishing write tid 1 to smithi17031214-36 2024-03-27T09:35:21.335 INFO:tasks.rados.rados.0.smithi170.stdout:209: done (4 left) 2024-03-27T09:35:21.335 INFO:tasks.rados.rados.0.smithi170.stdout:213: done (3 left) 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:214: done (2 left) 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:218: done (1 left) 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:219: read oid 16 snap -1 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:219: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:220: write (excl) oid 15 current snap is 15 2024-03-27T09:35:21.336 INFO:tasks.rados.rados.0.smithi170.stdout:220: seq_num 135 ranges {668945=536616,1633422=491363} 2024-03-27T09:35:21.342 INFO:tasks.rados.rados.0.smithi170.stdout:220: writing smithi17031214-15 from 668945 to 1205561 tid 1 2024-03-27T09:35:21.347 INFO:tasks.rados.rados.0.smithi170.stdout:220: writing smithi17031214-15 from 1633422 to 2124785 tid 2 2024-03-27T09:35:21.347 INFO:tasks.rados.rados.0.smithi170.stdout:221: snap_create 2024-03-27T09:35:21.353 INFO:tasks.rados.rados.0.smithi170.stdout:219: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-27T09:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:217: finishing write tid 2 to smithi17031214-36 2024-03-27T09:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:217: finishing write tid 3 to smithi17031214-36 2024-03-27T09:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:217: finishing write tid 4 to smithi17031214-36 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:217: oid 36 updating version 0 to 73 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:217: oid 36 updating version 73 to 75 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:217: oid 36 version 75 is already newer than 75 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 75 (ObjNum 133 snap 15 seq_num 133) dirty exists 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:217: left oid 36 (ObjNum 133 snap 15 seq_num 133) 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: finishing write tid 1 to smithi17031214-15 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: finishing write tid 2 to smithi17031214-15 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: finishing write tid 3 to smithi17031214-15 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: finishing write tid 4 to smithi17031214-15 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: oid 15 updating version 0 to 18 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: oid 15 version 18 is already newer than 17 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:220: oid 15 updating version 18 to 19 2024-03-27T09:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 19 (ObjNum 135 snap 15 seq_num 135) dirty exists 2024-03-27T09:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:220: left oid 15 (ObjNum 135 snap 15 seq_num 135) 2024-03-27T09:35:21.950 INFO:tasks.rados.rados.0.smithi170.stdout:217: done (3 left) 2024-03-27T09:35:21.951 INFO:tasks.rados.rados.0.smithi170.stdout:219: done (2 left) 2024-03-27T09:35:21.951 INFO:tasks.rados.rados.0.smithi170.stdout:220: done (1 left) 2024-03-27T09:35:21.951 INFO:tasks.rados.rados.0.smithi170.stdout:221: done (0 left) 2024-03-27T09:35:21.951 INFO:tasks.rados.rados.0.smithi170.stdout:222: snap_create 2024-03-27T09:35:22.953 INFO:tasks.rados.rados.0.smithi170.stdout:222: done (0 left) 2024-03-27T09:35:22.953 INFO:tasks.rados.rados.0.smithi170.stdout:223: write (excl) oid 45 current snap is 17 2024-03-27T09:35:22.954 INFO:tasks.rados.rados.0.smithi170.stdout:223: seq_num 136 ranges {685752=611865,1975551=310201} 2024-03-27T09:35:22.960 INFO:tasks.rados.rados.0.smithi170.stdout:223: writing smithi17031214-45 from 685752 to 1297617 tid 1 2024-03-27T09:35:22.966 INFO:tasks.rados.rados.0.smithi170.stdout:223: writing smithi17031214-45 from 1975551 to 2285752 tid 2 2024-03-27T09:35:22.966 INFO:tasks.rados.rados.0.smithi170.stdout:224: write oid 25 current snap is 17 2024-03-27T09:35:22.966 INFO:tasks.rados.rados.0.smithi170.stdout:224: seq_num 137 ranges {702559=403467,1750386=495032} 2024-03-27T09:35:22.972 INFO:tasks.rados.rados.0.smithi170.stdout:224: writing smithi17031214-25 from 702559 to 1106026 tid 1 2024-03-27T09:35:22.979 INFO:tasks.rados.rados.0.smithi170.stdout:224: writing smithi17031214-25 from 1750386 to 2245418 tid 2 2024-03-27T09:35:22.979 INFO:tasks.rados.rados.0.smithi170.stdout:225: write (excl) oid 19 current snap is 17 2024-03-27T09:35:22.979 INFO:tasks.rados.rados.0.smithi170.stdout:225: seq_num 138 ranges {719366=478716,1808868=510498} 2024-03-27T09:35:22.987 INFO:tasks.rados.rados.0.smithi170.stdout:225: writing smithi17031214-19 from 719366 to 1198082 tid 1 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:225: writing smithi17031214-19 from 1808868 to 2319366 tid 2 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:226: read oid 44 snap -1 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:227: read oid 32 snap -1 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:228: read oid 36 snap -1 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:228: expect (ObjNum 133 snap 15 seq_num 133) 2024-03-27T09:35:22.996 INFO:tasks.rados.rados.0.smithi170.stdout:229: read oid 6 snap -1 2024-03-27T09:35:22.997 INFO:tasks.rados.rados.0.smithi170.stdout:229: expect deleted 2024-03-27T09:35:22.997 INFO:tasks.rados.rados.0.smithi170.stdout:230: snap_remove snap 15 2024-03-27T09:35:23.413 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:23.523 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:23.520+0000 7fe16946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:23.526 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:23.526+0000 7fe16946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:23.958 INFO:tasks.rados.rados.0.smithi170.stdout:230: done (7 left) 2024-03-27T09:35:23.958 INFO:tasks.rados.rados.0.smithi170.stdout:231: read oid 31 snap -1 2024-03-27T09:35:23.958 INFO:tasks.rados.rados.0.smithi170.stdout:231: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-27T09:35:23.958 INFO:tasks.rados.rados.0.smithi170.stdout:232: write oid 2 current snap is 17 2024-03-27T09:35:23.959 INFO:tasks.rados.rados.0.smithi170.stdout:232: seq_num 139 ranges {736173=553965,1867350=468823} 2024-03-27T09:35:23.966 INFO:tasks.rados.rados.0.smithi170.stdout:232: writing smithi17031214-2 from 736173 to 1290138 tid 1 2024-03-27T09:35:23.973 INFO:tasks.rados.rados.0.smithi170.stdout:232: writing smithi17031214-2 from 1867350 to 2336173 tid 2 2024-03-27T09:35:23.973 INFO:tasks.rados.rados.0.smithi170.stdout:233: read oid 29 snap -1 2024-03-27T09:35:23.973 INFO:tasks.rados.rados.0.smithi170.stdout:233: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-27T09:35:23.973 INFO:tasks.rados.rados.0.smithi170.stdout:234: write oid 14 current snap is 17 2024-03-27T09:35:23.973 INFO:tasks.rados.rados.0.smithi170.stdout:234: seq_num 140 ranges {752980=629214,1809479=543501} 2024-03-27T09:35:23.981 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17031214-14 from 752980 to 1382194 tid 1 2024-03-27T09:35:23.989 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17031214-14 from 1809479 to 2352980 tid 2 2024-03-27T09:35:23.989 INFO:tasks.rados.rados.0.smithi170.stdout:235: delete oid 16 current snap is 17 2024-03-27T09:35:23.989 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 1 to smithi17031214-45 2024-03-27T09:35:23.989 INFO:tasks.rados.rados.0.smithi170.stdout:225: finishing write tid 1 to smithi17031214-19 2024-03-27T09:35:23.990 INFO:tasks.rados.rados.0.smithi170.stdout:227: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-27T09:35:24.101 INFO:tasks.rados.rados.0.smithi170.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:35:24.329 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 2 to smithi17031214-45 2024-03-27T09:35:24.329 INFO:tasks.rados.rados.0.smithi170.stdout:225: finishing write tid 2 to smithi17031214-19 2024-03-27T09:35:24.329 INFO:tasks.rados.rados.0.smithi170.stdout:228: expect (ObjNum 133 snap 15 seq_num 133) 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: finishing write tid 3 to smithi17031214-19 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: finishing write tid 4 to smithi17031214-19 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: oid 19 updating version 0 to 45 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: oid 19 updating version 45 to 47 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: oid 19 version 47 is already newer than 47 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 47 (ObjNum 138 snap 17 seq_num 138) dirty exists 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:225: left oid 19 (ObjNum 138 snap 17 seq_num 138) 2024-03-27T09:35:24.412 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 3 to smithi17031214-45 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 4 to smithi17031214-45 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 45 updating version 0 to 74 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 45 version 74 is already newer than 73 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 45 updating version 74 to 75 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 75 (ObjNum 136 snap 17 seq_num 136) dirty exists 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:223: left oid 45 (ObjNum 136 snap 17 seq_num 136) 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: finishing write tid 1 to smithi17031214-25 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: finishing write tid 2 to smithi17031214-25 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: finishing write tid 3 to smithi17031214-25 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: finishing write tid 4 to smithi17031214-25 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: oid 25 updating version 0 to 79 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: oid 25 version 79 is already newer than 79 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:224: oid 25 version 79 is already newer than 79 2024-03-27T09:35:24.413 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 79 (ObjNum 137 snap 17 seq_num 137) dirty exists 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:224: left oid 25 (ObjNum 137 snap 17 seq_num 137) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:223: done (11 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:224: done (10 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:225: done (9 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:226: done (8 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:227: done (7 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:228: done (6 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:229: done (5 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:235: done (4 left) 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:236: read oid 21 snap -1 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:236: expect deleted 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:237: delete oid 43 current snap is 17 2024-03-27T09:35:24.414 INFO:tasks.rados.rados.0.smithi170.stdout:232: finishing write tid 1 to smithi17031214-2 2024-03-27T09:35:24.415 INFO:tasks.rados.rados.0.smithi170.stdout:231: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-27T09:35:24.634 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 1 to smithi17031214-14 2024-03-27T09:35:24.635 INFO:tasks.rados.rados.0.smithi170.stdout:231: done (5 left) 2024-03-27T09:35:24.635 INFO:tasks.rados.rados.0.smithi170.stdout:237: done (4 left) 2024-03-27T09:35:24.635 INFO:tasks.rados.rados.0.smithi170.stdout:238: snap_remove snap 12 2024-03-27T09:35:24.960 INFO:tasks.rados.rados.0.smithi170.stdout:238: done (4 left) 2024-03-27T09:35:24.960 INFO:tasks.rados.rados.0.smithi170.stdout:239: read oid 43 snap -1 2024-03-27T09:35:24.961 INFO:tasks.rados.rados.0.smithi170.stdout:239: expect deleted 2024-03-27T09:35:24.961 INFO:tasks.rados.rados.0.smithi170.stdout:240: snap_remove snap 16 2024-03-27T09:35:25.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:25.229 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:25.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:35:25.340 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:25.336+0000 7f39f2a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:25.343 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:25.343+0000 7f39f2a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:25.528 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:35:25.528 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:35:25.541 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-27T09:35:24.958605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110507290,"num_objects":83,"num_object_clones":34,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":528,"num_read_kb":267568,"num_write":649,"num_write_kb":141911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":331236,"kb_used_data":222092,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377156124,"statfs":{"total":386547056640,"available":386207870976,"internally_reserved":0,"allocated":227422208,"data_stored":229738636,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2302328,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":48,"num_read_kb":45113,"num_write":39,"num_write_kb":8531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015883"},"pg_stats":[{"pgid":"3.f","version":"46'26","reported_seq":152,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.028815+0000","last_change":"2024-03-27T09:35:20.983301+0000","last_active":"2024-03-27T09:35:23.028815+0000","last_peered":"2024-03-27T09:35:23.028815+0000","last_clean":"2024-03-27T09:35:23.028815+0000","last_became_active":"2024-03-27T09:34:48.951324+0000","last_became_peered":"2024-03-27T09:34:48.951324+0000","last_unstale":"2024-03-27T09:35:23.028815+0000","last_undegraded":"2024-03-27T09:35:23.028815+0000","last_fullsized":"2024-03-27T09:35:23.028815+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563051,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":19472,"num_write":24,"num_write_kb":5444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"35'35","reported_seq":138,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.975255+0000","last_change":"2024-03-27T09:35:21.043936+0000","last_active":"2024-03-27T09:35:22.975255+0000","last_peered":"2024-03-27T09:35:22.975255+0000","last_clean":"2024-03-27T09:35:22.975255+0000","last_became_active":"2024-03-27T09:34:49.018740+0000","last_became_peered":"2024-03-27T09:34:49.018740+0000","last_unstale":"2024-03-27T09:35:22.975255+0000","last_undegraded":"2024-03-27T09:35:22.975255+0000","last_fullsized":"2024-03-27T09:35:22.975255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159683,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":10737,"num_write":28,"num_write_kb":6318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'47","reported_seq":161,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.059424+0000","last_change":"2024-03-27T09:35:21.029684+0000","last_active":"2024-03-27T09:35:23.059424+0000","last_peered":"2024-03-27T09:35:23.059424+0000","last_clean":"2024-03-27T09:35:23.059424+0000","last_became_active":"2024-03-27T09:34:48.937792+0000","last_became_peered":"2024-03-27T09:34:48.937792+0000","last_unstale":"2024-03-27T09:35:23.059424+0000","last_undegraded":"2024-03-27T09:35:23.059424+0000","last_fullsized":"2024-03-27T09:35:23.059424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521116,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":10330,"num_write":47,"num_write_kb":11992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"46'71","reported_seq":188,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.025852+0000","last_change":"2024-03-27T09:35:21.070175+0000","last_active":"2024-03-27T09:35:23.025852+0000","last_peered":"2024-03-27T09:35:23.025852+0000","last_clean":"2024-03-27T09:35:23.025852+0000","last_became_active":"2024-03-27T09:34:48.969295+0000","last_became_peered":"2024-03-27T09:34:48.969295+0000","last_unstale":"2024-03-27T09:35:23.025852+0000","last_undegraded":"2024-03-27T09:35:23.025852+0000","last_fullsized":"2024-03-27T09:35:23.025852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11613538,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":49,"num_read_kb":28039,"num_write":60,"num_write_kb":16920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"45'5","reported_seq":107,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.022089+0000","last_change":"2024-03-27T09:35:21.052854+0000","last_active":"2024-03-27T09:35:23.022089+0000","last_peered":"2024-03-27T09:35:23.022089+0000","last_clean":"2024-03-27T09:35:23.022089+0000","last_became_active":"2024-03-27T09:34:49.019732+0000","last_became_peered":"2024-03-27T09:34:49.019732+0000","last_unstale":"2024-03-27T09:35:23.022089+0000","last_undegraded":"2024-03-27T09:35:23.022089+0000","last_fullsized":"2024-03-27T09:35:23.022089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"37'51","reported_seq":190,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.053973+0000","last_change":"2024-03-27T09:35:20.995879+0000","last_active":"2024-03-27T09:35:23.053973+0000","last_peered":"2024-03-27T09:35:23.053973+0000","last_clean":"2024-03-27T09:35:23.053973+0000","last_became_active":"2024-03-27T09:34:48.976415+0000","last_became_peered":"2024-03-27T09:34:48.976415+0000","last_unstale":"2024-03-27T09:35:23.053973+0000","last_undegraded":"2024-03-27T09:35:23.053973+0000","last_fullsized":"2024-03-27T09:35:23.053973+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19798448,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":38634,"num_write":53,"num_write_kb":16285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"46'38","reported_seq":165,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.028265+0000","last_change":"2024-03-27T09:35:21.068204+0000","last_active":"2024-03-27T09:35:23.028265+0000","last_peered":"2024-03-27T09:35:23.028265+0000","last_clean":"2024-03-27T09:35:23.028265+0000","last_became_active":"2024-03-27T09:34:48.971193+0000","last_became_peered":"2024-03-27T09:34:48.971193+0000","last_unstale":"2024-03-27T09:35:23.028265+0000","last_undegraded":"2024-03-27T09:35:23.028265+0000","last_fullsized":"2024-03-27T09:35:23.028265+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7428661,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":25837,"num_write":36,"num_write_kb":10238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"48'79","reported_seq":204,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.111813+0000","last_change":"2024-03-27T09:35:21.036211+0000","last_active":"2024-03-27T09:35:23.111813+0000","last_peered":"2024-03-27T09:35:23.111813+0000","last_clean":"2024-03-27T09:35:23.111813+0000","last_became_active":"2024-03-27T09:34:49.005787+0000","last_became_peered":"2024-03-27T09:34:49.005787+0000","last_unstale":"2024-03-27T09:35:23.111813+0000","last_undegraded":"2024-03-27T09:35:23.111813+0000","last_fullsized":"2024-03-27T09:35:23.111813+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14739640,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":38072,"num_write":81,"num_write_kb":23915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":225,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.012668+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:35:23.012668+0000","last_peered":"2024-03-27T09:35:23.012668+0000","last_clean":"2024-03-27T09:35:23.012668+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:35:23.012668+0000","last_undegraded":"2024-03-27T09:35:23.012668+0000","last_fullsized":"2024-03-27T09:35:23.012668+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.997744+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:35:22.997744+0000","last_peered":"2024-03-27T09:35:22.997744+0000","last_clean":"2024-03-27T09:35:22.997744+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:35:22.997744+0000","last_undegraded":"2024-03-27T09:35:22.997744+0000","last_fullsized":"2024-03-27T09:35:22.997744+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":108,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.994374+0000","last_change":"2024-03-27T09:35:21.035698+0000","last_active":"2024-03-27T09:35:22.994374+0000","last_peered":"2024-03-27T09:35:22.994374+0000","last_clean":"2024-03-27T09:35:22.994374+0000","last_became_active":"2024-03-27T09:34:48.987725+0000","last_became_peered":"2024-03-27T09:34:48.987725+0000","last_unstale":"2024-03-27T09:35:22.994374+0000","last_undegraded":"2024-03-27T09:35:22.994374+0000","last_fullsized":"2024-03-27T09:35:22.994374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":109,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.027331+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:35:23.027331+0000","last_peered":"2024-03-27T09:35:23.027331+0000","last_clean":"2024-03-27T09:35:23.027331+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:35:23.027331+0000","last_undegraded":"2024-03-27T09:35:23.027331+0000","last_fullsized":"2024-03-27T09:35:23.027331+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'7","reported_seq":113,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.025036+0000","last_change":"2024-03-27T09:35:21.049436+0000","last_active":"2024-03-27T09:35:23.025036+0000","last_peered":"2024-03-27T09:35:23.025036+0000","last_clean":"2024-03-27T09:35:23.025036+0000","last_became_active":"2024-03-27T09:34:49.047522+0000","last_became_peered":"2024-03-27T09:34:49.047522+0000","last_unstale":"2024-03-27T09:35:23.025036+0000","last_undegraded":"2024-03-27T09:35:23.025036+0000","last_fullsized":"2024-03-27T09:35:23.025036+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3966419,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4008,"num_write":9,"num_write_kb":2512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.985287+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:35:22.985287+0000","last_peered":"2024-03-27T09:35:22.985287+0000","last_clean":"2024-03-27T09:35:22.985287+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:35:22.985287+0000","last_undegraded":"2024-03-27T09:35:22.985287+0000","last_fullsized":"2024-03-27T09:35:22.985287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"46'20","reported_seq":132,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.026874+0000","last_change":"2024-03-27T09:35:21.086983+0000","last_active":"2024-03-27T09:35:23.026874+0000","last_peered":"2024-03-27T09:35:23.026874+0000","last_clean":"2024-03-27T09:35:23.026874+0000","last_became_active":"2024-03-27T09:34:48.981602+0000","last_became_peered":"2024-03-27T09:34:48.981602+0000","last_unstale":"2024-03-27T09:35:23.026874+0000","last_undegraded":"2024-03-27T09:35:23.026874+0000","last_fullsized":"2024-03-27T09:35:23.026874+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4453822,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":27298,"num_write":16,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.025567+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:35:23.025567+0000","last_peered":"2024-03-27T09:35:23.025567+0000","last_clean":"2024-03-27T09:35:23.025567+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:35:23.025567+0000","last_undegraded":"2024-03-27T09:35:23.025567+0000","last_fullsized":"2024-03-27T09:35:23.025567+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"46'21","reported_seq":127,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.967992+0000","last_change":"2024-03-27T09:35:21.068838+0000","last_active":"2024-03-27T09:35:22.967992+0000","last_peered":"2024-03-27T09:35:22.967992+0000","last_clean":"2024-03-27T09:35:22.967992+0000","last_became_active":"2024-03-27T09:34:48.945926+0000","last_became_peered":"2024-03-27T09:34:48.945926+0000","last_unstale":"2024-03-27T09:35:22.967992+0000","last_undegraded":"2024-03-27T09:35:22.967992+0000","last_fullsized":"2024-03-27T09:35:22.967992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8722767,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3051,"num_write":21,"num_write_kb":6018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"37'38","reported_seq":161,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.989478+0000","last_change":"2024-03-27T09:35:21.029389+0000","last_active":"2024-03-27T09:35:22.989478+0000","last_peered":"2024-03-27T09:35:22.989478+0000","last_clean":"2024-03-27T09:35:22.989478+0000","last_became_active":"2024-03-27T09:34:48.958003+0000","last_became_peered":"2024-03-27T09:34:48.958003+0000","last_unstale":"2024-03-27T09:35:22.989478+0000","last_undegraded":"2024-03-27T09:35:22.989478+0000","last_fullsized":"2024-03-27T09:35:22.989478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966452,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":20272,"num_write":44,"num_write_kb":10935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":97,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.971843+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:35:22.971843+0000","last_peered":"2024-03-27T09:35:22.971843+0000","last_clean":"2024-03-27T09:35:22.971843+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:35:22.971843+0000","last_undegraded":"2024-03-27T09:35:22.971843+0000","last_fullsized":"2024-03-27T09:35:22.971843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"46'19","reported_seq":129,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.972022+0000","last_change":"2024-03-27T09:35:21.085130+0000","last_active":"2024-03-27T09:35:22.972022+0000","last_peered":"2024-03-27T09:35:22.972022+0000","last_clean":"2024-03-27T09:35:22.972022+0000","last_became_active":"2024-03-27T09:34:48.995086+0000","last_became_peered":"2024-03-27T09:34:48.995086+0000","last_unstale":"2024-03-27T09:35:22.972022+0000","last_undegraded":"2024-03-27T09:35:22.972022+0000","last_fullsized":"2024-03-27T09:35:22.972022+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4655506,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":6199,"num_write":21,"num_write_kb":5276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.033063+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:35:23.033063+0000","last_peered":"2024-03-27T09:35:23.033063+0000","last_clean":"2024-03-27T09:35:23.033063+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:35:23.033063+0000","last_undegraded":"2024-03-27T09:35:23.033063+0000","last_fullsized":"2024-03-27T09:35:23.033063+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"46'75","reported_seq":187,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:23.076872+0000","last_change":"2024-03-27T09:35:21.092008+0000","last_active":"2024-03-27T09:35:23.076872+0000","last_peered":"2024-03-27T09:35:23.076872+0000","last_clean":"2024-03-27T09:35:23.076872+0000","last_became_active":"2024-03-27T09:34:49.013275+0000","last_became_peered":"2024-03-27T09:34:49.013275+0000","last_unstale":"2024-03-27T09:35:23.076872+0000","last_undegraded":"2024-03-27T09:35:23.076872+0000","last_fullsized":"2024-03-27T09:35:23.076872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9327819,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":42,"num_read_kb":35527,"num_write":67,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.974324+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:35:22.974324+0000","last_peered":"2024-03-27T09:35:22.974324+0000","last_clean":"2024-03-27T09:35:22.974324+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:35:22.974324+0000","last_undegraded":"2024-03-27T09:35:22.974324+0000","last_fullsized":"2024-03-27T09:35:22.974324+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.963756+0000","last_change":"2024-03-27T09:35:21.023854+0000","last_active":"2024-03-27T09:35:22.963756+0000","last_peered":"2024-03-27T09:35:22.963756+0000","last_clean":"2024-03-27T09:35:22.963756+0000","last_became_active":"2024-03-27T09:34:48.967426+0000","last_became_peered":"2024-03-27T09:34:48.967426+0000","last_unstale":"2024-03-27T09:35:22.963756+0000","last_undegraded":"2024-03-27T09:35:22.963756+0000","last_fullsized":"2024-03-27T09:35:22.963756+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":103,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-27T09:35:22.995287+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:35:22.995287+0000","last_peered":"2024-03-27T09:35:22.995287+0000","last_clean":"2024-03-27T09:35:22.995287+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:35:22.995287+0000","last_undegraded":"2024-03-27T09:35:22.995287+0000","last_fullsized":"2024-03-27T09:35:22.995287+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":109916922,"num_objects":81,"num_object_clones":34,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":425,"num_read_kb":267477,"num_write":512,"num_write_kb":140017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":532,"ondisk_log_size":532,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85260,"kb_used_data":57820,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94286580,"statfs":{"total":96636764160,"available":96549457920,"internally_reserved":0,"allocated":59207680,"data_stored":60102815,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85504,"kb_used_data":58064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94286336,"statfs":{"total":96636764160,"available":96549208064,"internally_reserved":0,"allocated":59457536,"data_stored":60432801,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81660,"kb_used_data":54836,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94290180,"statfs":{"total":96636764160,"available":96553144320,"internally_reserved":0,"allocated":56152064,"data_stored":55987903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78812,"kb_used_data":51372,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293028,"statfs":{"total":96636764160,"available":96556060672,"internally_reserved":0,"allocated":52604928,"data_stored":53215117,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:35:25.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:25.747 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:25.745+0000 7f292c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:25.752 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:25.751+0000 7f292c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:25.966 INFO:tasks.rados.rados.0.smithi170.stdout:240: done (5 left) 2024-03-27T09:35:25.966 INFO:tasks.rados.rados.0.smithi170.stdout:241: read oid 26 snap -1 2024-03-27T09:35:25.966 INFO:tasks.rados.rados.0.smithi170.stdout:241: expect (ObjNum 120 snap 10 seq_num 120) 2024-03-27T09:35:25.966 INFO:tasks.rados.rados.0.smithi170.stdout:242: read oid 11 snap -1 2024-03-27T09:35:25.966 INFO:tasks.rados.rados.0.smithi170.stdout:242: expect deleted 2024-03-27T09:35:25.967 INFO:tasks.rados.rados.0.smithi170.stdout:243: snap_create 2024-03-27T09:35:25.967 INFO:tasks.rados.rados.0.smithi170.stdout:232: finishing write tid 2 to smithi17031214-2 2024-03-27T09:35:25.967 INFO:tasks.rados.rados.0.smithi170.stdout:233: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: finishing write tid 3 to smithi17031214-2 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: finishing write tid 4 to smithi17031214-2 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: oid 2 updating version 0 to 85 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: oid 2 version 85 is already newer than 84 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: oid 2 version 85 is already newer than 84 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 85 (ObjNum 139 snap 17 seq_num 139) dirty exists 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:232: left oid 2 (ObjNum 139 snap 17 seq_num 139) 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 2 to smithi17031214-14 2024-03-27T09:35:26.070 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 3 to smithi17031214-14 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 4 to smithi17031214-14 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 14 updating version 0 to 29 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 14 version 29 is already newer than 28 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 14 updating version 29 to 30 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 30 (ObjNum 140 snap 17 seq_num 140) dirty exists 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:234: left oid 14 (ObjNum 140 snap 17 seq_num 140) 2024-03-27T09:35:26.071 INFO:tasks.rados.rados.0.smithi170.stdout:241: expect (ObjNum 120 snap 10 seq_num 120) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:232: done (7 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:233: done (6 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:234: done (5 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:236: done (4 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:239: done (3 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:241: done (2 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:242: done (1 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:243: done (0 left) 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:244: write (excl) oid 49 current snap is 18 2024-03-27T09:35:26.968 INFO:tasks.rados.rados.0.smithi170.stdout:244: seq_num 143 ranges {403401=454961,1584925=506039} 2024-03-27T09:35:26.972 INFO:tasks.rados.rados.0.smithi170.stdout:244: writing smithi17031214-49 from 403401 to 858362 tid 1 2024-03-27T09:35:26.979 INFO:tasks.rados.rados.0.smithi170.stdout:244: writing smithi17031214-49 from 1584925 to 2090964 tid 2 2024-03-27T09:35:26.979 INFO:tasks.rados.rados.0.smithi170.stdout:245: snap_create 2024-03-27T09:35:27.057 INFO:tasks.rados.rados.0.smithi170.stdout:244: finishing write tid 1 to smithi17031214-49 2024-03-27T09:35:27.080 INFO:tasks.rados.rados.0.smithi170.stdout:244: finishing write tid 2 to smithi17031214-49 2024-03-27T09:35:27.087 INFO:tasks.rados.rados.0.smithi170.stdout:244: finishing write tid 3 to smithi17031214-49 2024-03-27T09:35:27.088 INFO:tasks.rados.rados.0.smithi170.stdout:244: finishing write tid 4 to smithi17031214-49 2024-03-27T09:35:27.088 INFO:tasks.rados.rados.0.smithi170.stdout:244: oid 49 updating version 0 to 44 2024-03-27T09:35:27.088 INFO:tasks.rados.rados.0.smithi170.stdout:244: oid 49 version 44 is already newer than 44 2024-03-27T09:35:27.088 INFO:tasks.rados.rados.0.smithi170.stdout:244: oid 49 version 44 is already newer than 42 2024-03-27T09:35:27.088 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 44 (ObjNum 143 snap 18 seq_num 143) dirty exists 2024-03-27T09:35:27.089 INFO:tasks.rados.rados.0.smithi170.stdout:244: left oid 49 (ObjNum 143 snap 18 seq_num 143) 2024-03-27T09:35:27.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:244: done (1 left) 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:245: done (0 left) 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:246: read oid 49 snap -1 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:247: write oid 37 current snap is 19 2024-03-27T09:35:27.972 INFO:tasks.rados.rados.0.smithi170.stdout:247: seq_num 144 ranges {420208=530210,1527054=766050} 2024-03-27T09:35:27.979 INFO:tasks.rados.rados.0.smithi170.stdout:247: writing smithi17031214-37 from 420208 to 950418 tid 1 2024-03-27T09:35:27.990 INFO:tasks.rados.rados.0.smithi170.stdout:247: writing smithi17031214-37 from 1527054 to 2293104 tid 2 2024-03-27T09:35:27.990 INFO:tasks.rados.rados.0.smithi170.stdout:248: write oid 48 current snap is 19 2024-03-27T09:35:27.990 INFO:tasks.rados.rados.0.smithi170.stdout:248: seq_num 145 ranges {437015=721812,1701889=509708} 2024-03-27T09:35:27.999 INFO:tasks.rados.rados.0.smithi170.stdout:248: writing smithi17031214-48 from 437015 to 1158827 tid 1 2024-03-27T09:35:28.007 INFO:tasks.rados.rados.0.smithi170.stdout:248: writing smithi17031214-48 from 1701889 to 2211597 tid 2 2024-03-27T09:35:28.008 INFO:tasks.rados.rados.0.smithi170.stdout:249: snap_create 2024-03-27T09:35:28.068 INFO:tasks.rados.rados.0.smithi170.stdout:247: finishing write tid 1 to smithi17031214-37 2024-03-27T09:35:28.068 INFO:tasks.rados.rados.0.smithi170.stdout:246: expect (ObjNum 143 snap 18 seq_num 143) 2024-03-27T09:35:28.069 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:28.066+0000 7fb00cdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:28.078 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:28.077+0000 7fb00cdfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 1 to smithi17031214-48 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 2 to smithi17031214-48 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: finishing write tid 2 to smithi17031214-37 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: finishing write tid 3 to smithi17031214-37 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 3 to smithi17031214-48 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: finishing write tid 4 to smithi17031214-37 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: oid 37 updating version 0 to 55 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: oid 37 version 55 is already newer than 55 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: oid 37 version 55 is already newer than 53 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 55 (ObjNum 144 snap 19 seq_num 144) dirty exists 2024-03-27T09:35:28.210 INFO:tasks.rados.rados.0.smithi170.stdout:247: left oid 37 (ObjNum 144 snap 19 seq_num 144) 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 4 to smithi17031214-48 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 48 updating version 0 to 45 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 48 updating version 45 to 46 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 48 updating version 46 to 47 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 47 (ObjNum 145 snap 19 seq_num 145) dirty exists 2024-03-27T09:35:28.211 INFO:tasks.rados.rados.0.smithi170.stdout:248: left oid 48 (ObjNum 145 snap 19 seq_num 145) 2024-03-27T09:35:28.976 INFO:tasks.rados.rados.0.smithi170.stdout:246: done (3 left) 2024-03-27T09:35:28.976 INFO:tasks.rados.rados.0.smithi170.stdout:247: done (2 left) 2024-03-27T09:35:28.976 INFO:tasks.rados.rados.0.smithi170.stdout:248: done (1 left) 2024-03-27T09:35:28.977 INFO:tasks.rados.rados.0.smithi170.stdout:249: done (0 left) 2024-03-27T09:35:28.977 INFO:tasks.rados.rados.0.smithi170.stdout:250: read oid 44 snap -1 2024-03-27T09:35:28.977 INFO:tasks.rados.rados.0.smithi170.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:35:28.977 INFO:tasks.rados.rados.0.smithi170.stdout:251: write (excl) oid 32 current snap is 20 2024-03-27T09:35:28.977 INFO:tasks.rados.rados.0.smithi170.stdout:251: seq_num 146 ranges {453822=797061,1760371=653366} 2024-03-27T09:35:28.986 INFO:tasks.rados.rados.0.smithi170.stdout:251: writing smithi17031214-32 from 453822 to 1250883 tid 1 2024-03-27T09:35:28.996 INFO:tasks.rados.rados.0.smithi170.stdout:251: writing smithi17031214-32 from 1760371 to 2413737 tid 2 2024-03-27T09:35:28.996 INFO:tasks.rados.rados.0.smithi170.stdout:252: write oid 10 current snap is 20 2024-03-27T09:35:28.996 INFO:tasks.rados.rados.0.smithi170.stdout:252: seq_num 147 ranges {470629=472310,1418853=513377} 2024-03-27T09:35:29.003 INFO:tasks.rados.rados.0.smithi170.stdout:252: writing smithi17031214-10 from 470629 to 942939 tid 1 2024-03-27T09:35:29.011 INFO:tasks.rados.rados.0.smithi170.stdout:252: writing smithi17031214-10 from 1418853 to 1932230 tid 2 2024-03-27T09:35:29.011 INFO:tasks.rados.rados.0.smithi170.stdout:253: delete oid 46 current snap is 20 2024-03-27T09:35:29.040 INFO:tasks.rados.rados.0.smithi170.stdout:253: done (3 left) 2024-03-27T09:35:29.040 INFO:tasks.rados.rados.0.smithi170.stdout:254: snap_remove snap 19 2024-03-27T09:35:29.978 INFO:tasks.rados.rados.0.smithi170.stdout:254: done (3 left) 2024-03-27T09:35:29.978 INFO:tasks.rados.rados.0.smithi170.stdout:255: delete oid 43 current snap is 20 2024-03-27T09:35:29.978 INFO:tasks.rados.rados.0.smithi170.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: finishing write tid 1 to smithi17031214-32 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:252: finishing write tid 1 to smithi17031214-10 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: finishing write tid 2 to smithi17031214-32 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:252: finishing write tid 2 to smithi17031214-10 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: finishing write tid 3 to smithi17031214-32 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: finishing write tid 4 to smithi17031214-32 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: oid 32 updating version 0 to 59 2024-03-27T09:35:30.213 INFO:tasks.rados.rados.0.smithi170.stdout:251: oid 32 version 59 is already newer than 57 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:251: oid 32 version 59 is already newer than 58 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 59 (ObjNum 146 snap 20 seq_num 146) dirty exists 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:251: left oid 32 (ObjNum 146 snap 20 seq_num 146) 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:252: finishing write tid 3 to smithi17031214-10 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:250: done (3 left) 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:251: done (2 left) 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:255: done (1 left) 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:256: read oid 50 snap 18 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:256: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:257: write (excl) oid 32 current snap is 20 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:252: finishing write tid 4 to smithi17031214-10 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:252: oid 10 updating version 0 to 7 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:252: oid 10 updating version 7 to 8 2024-03-27T09:35:30.214 INFO:tasks.rados.rados.0.smithi170.stdout:252: oid 10 version 8 is already newer than 6 2024-03-27T09:35:30.215 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 8 (ObjNum 147 snap 20 seq_num 147) dirty exists 2024-03-27T09:35:30.215 INFO:tasks.rados.rados.0.smithi170.stdout:252: left oid 10 (ObjNum 147 snap 20 seq_num 147) 2024-03-27T09:35:30.215 INFO:tasks.rados.rados.0.smithi170.stdout:257: seq_num 150 ranges {521050=698057,1877946=643104} 2024-03-27T09:35:30.222 INFO:tasks.rados.rados.0.smithi170.stdout:257: writing smithi17031214-32 from 521050 to 1219107 tid 1 2024-03-27T09:35:30.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:30.230 INFO:tasks.rados.rados.0.smithi170.stdout:257: writing smithi17031214-32 from 1877946 to 2521050 tid 2 2024-03-27T09:35:30.230 INFO:tasks.rados.rados.0.smithi170.stdout:252: done (2 left) 2024-03-27T09:35:30.230 INFO:tasks.rados.rados.0.smithi170.stdout:258: write oid 29 current snap is 20 2024-03-27T09:35:30.230 INFO:tasks.rados.rados.0.smithi170.stdout:258: seq_num 151 ranges {537857=773306,1936428=520715} 2024-03-27T09:35:30.237 INFO:tasks.rados.rados.0.smithi170.stdout:258: writing smithi17031214-29 from 537857 to 1311163 tid 1 2024-03-27T09:35:30.245 INFO:tasks.rados.rados.0.smithi170.stdout:258: writing smithi17031214-29 from 1936428 to 2457143 tid 2 2024-03-27T09:35:30.246 INFO:tasks.rados.rados.0.smithi170.stdout:259: snap_create 2024-03-27T09:35:30.279 INFO:tasks.rados.rados.0.smithi170.stdout:256: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-27T09:35:30.372 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:30.369+0000 7f3f8af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:30.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:30.375+0000 7f3f8af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:30.467 INFO:tasks.rados.rados.0.smithi170.stdout:258: finishing write tid 1 to smithi17031214-29 2024-03-27T09:35:30.467 INFO:tasks.rados.rados.0.smithi170.stdout:258: finishing write tid 2 to smithi17031214-29 2024-03-27T09:35:30.467 INFO:tasks.rados.rados.0.smithi170.stdout:257: finishing write tid 1 to smithi17031214-32 2024-03-27T09:35:30.467 INFO:tasks.rados.rados.0.smithi170.stdout:258: finishing write tid 3 to smithi17031214-29 2024-03-27T09:35:30.467 INFO:tasks.rados.rados.0.smithi170.stdout:258: finishing write tid 4 to smithi17031214-29 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:258: oid 29 updating version 0 to 42 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:258: oid 29 version 42 is already newer than 41 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:258: oid 29 version 42 is already newer than 40 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 42 (ObjNum 151 snap 20 seq_num 151) dirty exists 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:258: left oid 29 (ObjNum 151 snap 20 seq_num 151) 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: finishing write tid 2 to smithi17031214-32 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: finishing write tid 3 to smithi17031214-32 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: finishing write tid 4 to smithi17031214-32 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: oid 32 updating version 0 to 64 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: oid 32 version 64 is already newer than 64 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: oid 32 version 64 is already newer than 62 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 64 (ObjNum 150 snap 20 seq_num 150) dirty exists 2024-03-27T09:35:30.468 INFO:tasks.rados.rados.0.smithi170.stdout:257: left oid 32 (ObjNum 150 snap 20 seq_num 150) 2024-03-27T09:35:30.543 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:30.543 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:30.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T09:35:30.654 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:30.649+0000 7f6ed3d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:30.657 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:30.657+0000 7f6ed3d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:30.850 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-27T09:35:30.850 INFO:teuthology.orchestra.run.smithi170.stderr:dumped all 2024-03-27T09:35:30.863 INFO:teuthology.orchestra.run.smithi170.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-27T09:35:29.976469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104204863,"num_objects":90,"num_object_clones":40,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":558,"num_read_kb":290488,"num_write":671,"num_write_kb":147466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":694,"ondisk_log_size":694,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":279304,"kb_used_data":172008,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377208056,"statfs":{"total":386547056640,"available":386261049344,"internally_reserved":0,"allocated":176136192,"data_stored":177305708,"data_compressed":89751,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6302427,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":30,"num_read_kb":22920,"num_write":22,"num_write_kb":5555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017060"},"pg_stats":[{"pgid":"3.f","version":"52'36","reported_seq":172,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.998757+0000","last_change":"2024-03-27T09:35:27.045396+0000","last_active":"2024-03-27T09:35:27.998757+0000","last_peered":"2024-03-27T09:35:27.998757+0000","last_clean":"2024-03-27T09:35:27.998757+0000","last_became_active":"2024-03-27T09:34:48.951324+0000","last_became_peered":"2024-03-27T09:34:48.951324+0000","last_unstale":"2024-03-27T09:35:27.998757+0000","last_undegraded":"2024-03-27T09:35:27.998757+0000","last_fullsized":"2024-03-27T09:35:27.998757+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2352980,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":37,"num_read_kb":19473,"num_write":28,"num_write_kb":6590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"35'35","reported_seq":144,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.790749+0000","last_change":"2024-03-27T09:35:25.815664+0000","last_active":"2024-03-27T09:35:25.790749+0000","last_peered":"2024-03-27T09:35:25.790749+0000","last_clean":"2024-03-27T09:35:25.790749+0000","last_became_active":"2024-03-27T09:34:49.018740+0000","last_became_peered":"2024-03-27T09:34:49.018740+0000","last_unstale":"2024-03-27T09:35:25.790749+0000","last_undegraded":"2024-03-27T09:35:25.790749+0000","last_fullsized":"2024-03-27T09:35:25.790749+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5159683,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":16,"num_read_kb":10737,"num_write":28,"num_write_kb":6318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"48'47","reported_seq":176,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:26.034048+0000","last_change":"2024-03-27T09:35:25.983016+0000","last_active":"2024-03-27T09:35:26.034048+0000","last_peered":"2024-03-27T09:35:26.034048+0000","last_clean":"2024-03-27T09:35:26.034048+0000","last_became_active":"2024-03-27T09:34:48.937792+0000","last_became_peered":"2024-03-27T09:34:48.937792+0000","last_unstale":"2024-03-27T09:35:26.034048+0000","last_undegraded":"2024-03-27T09:35:26.034048+0000","last_fullsized":"2024-03-27T09:35:26.034048+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521116,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":14716,"num_write":47,"num_write_kb":11992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"50'75","reported_seq":204,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.997127+0000","last_change":"2024-03-27T09:35:27.014079+0000","last_active":"2024-03-27T09:35:27.997127+0000","last_peered":"2024-03-27T09:35:27.997127+0000","last_clean":"2024-03-27T09:35:27.997127+0000","last_became_active":"2024-03-27T09:34:48.969295+0000","last_became_peered":"2024-03-27T09:34:48.969295+0000","last_unstale":"2024-03-27T09:35:27.997127+0000","last_undegraded":"2024-03-27T09:35:27.997127+0000","last_fullsized":"2024-03-27T09:35:27.997127+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11613538,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":49,"num_read_kb":28039,"num_write":60,"num_write_kb":16920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"45'5","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.001182+0000","last_change":"2024-03-27T09:35:27.009342+0000","last_active":"2024-03-27T09:35:27.001182+0000","last_peered":"2024-03-27T09:35:27.001182+0000","last_clean":"2024-03-27T09:35:27.001182+0000","last_became_active":"2024-03-27T09:34:49.019732+0000","last_became_peered":"2024-03-27T09:34:49.019732+0000","last_unstale":"2024-03-27T09:35:27.001182+0000","last_undegraded":"2024-03-27T09:35:27.001182+0000","last_fullsized":"2024-03-27T09:35:27.001182+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"53'55","reported_seq":210,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:28.071148+0000","last_change":"2024-03-27T09:35:27.016875+0000","last_active":"2024-03-27T09:35:28.071148+0000","last_peered":"2024-03-27T09:35:28.071148+0000","last_clean":"2024-03-27T09:35:28.071148+0000","last_became_active":"2024-03-27T09:34:48.976415+0000","last_became_peered":"2024-03-27T09:34:48.976415+0000","last_unstale":"2024-03-27T09:35:28.071148+0000","last_undegraded":"2024-03-27T09:35:28.071148+0000","last_fullsized":"2024-03-27T09:35:28.071148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19613604,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":38635,"num_write":57,"num_write_kb":17552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"53'47","reported_seq":192,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:28.108798+0000","last_change":"2024-03-27T09:35:27.017512+0000","last_active":"2024-03-27T09:35:28.108798+0000","last_peered":"2024-03-27T09:35:28.108798+0000","last_clean":"2024-03-27T09:35:28.108798+0000","last_became_active":"2024-03-27T09:34:48.971193+0000","last_became_peered":"2024-03-27T09:34:48.971193+0000","last_unstale":"2024-03-27T09:35:28.108798+0000","last_undegraded":"2024-03-27T09:35:28.108798+0000","last_fullsized":"2024-03-27T09:35:28.108798+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8924517,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":50,"num_read_kb":32231,"num_write":44,"num_write_kb":12381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"51'101","reported_seq":221,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.986600+0000","last_change":"2024-03-27T09:35:26.056193+0000","last_active":"2024-03-27T09:35:25.986600+0000","last_peered":"2024-03-27T09:35:25.986600+0000","last_clean":"2024-03-27T09:35:25.986600+0000","last_became_active":"2024-03-27T09:34:49.005787+0000","last_became_peered":"2024-03-27T09:34:49.005787+0000","last_unstale":"2024-03-27T09:35:25.986600+0000","last_undegraded":"2024-03-27T09:35:25.986600+0000","last_fullsized":"2024-03-27T09:35:25.986600+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10588377,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":38073,"num_write":86,"num_write_kb":24914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:26.982710+0000","last_change":"2024-03-27T09:34:41.790364+0000","last_active":"2024-03-27T09:35:26.982710+0000","last_peered":"2024-03-27T09:35:26.982710+0000","last_clean":"2024-03-27T09:35:26.982710+0000","last_became_active":"2024-03-27T09:34:41.782057+0000","last_became_peered":"2024-03-27T09:34:41.782057+0000","last_unstale":"2024-03-27T09:35:26.982710+0000","last_undegraded":"2024-03-27T09:35:26.982710+0000","last_fullsized":"2024-03-27T09:35:26.982710+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:40.721203+0000","last_clean_scrub_stamp":"2024-03-27T09:34:40.721203+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.985481+0000","last_change":"2024-03-27T09:34:42.850569+0000","last_active":"2024-03-27T09:35:25.985481+0000","last_peered":"2024-03-27T09:35:25.985481+0000","last_clean":"2024-03-27T09:35:25.985481+0000","last_became_active":"2024-03-27T09:34:42.844530+0000","last_became_peered":"2024-03-27T09:34:42.844530+0000","last_unstale":"2024-03-27T09:35:25.985481+0000","last_undegraded":"2024-03-27T09:35:25.985481+0000","last_fullsized":"2024-03-27T09:35:25.985481+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":120,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.982225+0000","last_change":"2024-03-27T09:35:25.989474+0000","last_active":"2024-03-27T09:35:25.982225+0000","last_peered":"2024-03-27T09:35:25.982225+0000","last_clean":"2024-03-27T09:35:25.982225+0000","last_became_active":"2024-03-27T09:34:48.987725+0000","last_became_peered":"2024-03-27T09:34:48.987725+0000","last_unstale":"2024-03-27T09:35:25.982225+0000","last_undegraded":"2024-03-27T09:35:25.982225+0000","last_fullsized":"2024-03-27T09:35:25.982225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":122,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:28.000342+0000","last_change":"2024-03-27T09:34:42.827269+0000","last_active":"2024-03-27T09:35:28.000342+0000","last_peered":"2024-03-27T09:35:28.000342+0000","last_clean":"2024-03-27T09:35:28.000342+0000","last_became_active":"2024-03-27T09:34:42.822003+0000","last_became_peered":"2024-03-27T09:34:42.822003+0000","last_unstale":"2024-03-27T09:35:28.000342+0000","last_undegraded":"2024-03-27T09:35:28.000342+0000","last_fullsized":"2024-03-27T09:35:28.000342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"37'7","reported_seq":126,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.002534+0000","last_change":"2024-03-27T09:35:27.009590+0000","last_active":"2024-03-27T09:35:27.002534+0000","last_peered":"2024-03-27T09:35:27.002534+0000","last_clean":"2024-03-27T09:35:27.002534+0000","last_became_active":"2024-03-27T09:34:49.047522+0000","last_became_peered":"2024-03-27T09:34:49.047522+0000","last_unstale":"2024-03-27T09:35:27.002534+0000","last_undegraded":"2024-03-27T09:35:27.002534+0000","last_fullsized":"2024-03-27T09:35:27.002534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3966419,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4008,"num_write":9,"num_write_kb":2512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":112,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.977118+0000","last_change":"2024-03-27T09:34:42.830207+0000","last_active":"2024-03-27T09:35:25.977118+0000","last_peered":"2024-03-27T09:35:25.977118+0000","last_clean":"2024-03-27T09:35:25.977118+0000","last_became_active":"2024-03-27T09:34:42.818784+0000","last_became_peered":"2024-03-27T09:34:42.818784+0000","last_unstale":"2024-03-27T09:35:25.977118+0000","last_undegraded":"2024-03-27T09:35:25.977118+0000","last_fullsized":"2024-03-27T09:35:25.977118+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"52'28","reported_seq":146,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.000221+0000","last_change":"2024-03-27T09:35:27.028446+0000","last_active":"2024-03-27T09:35:27.000221+0000","last_peered":"2024-03-27T09:35:27.000221+0000","last_clean":"2024-03-27T09:35:27.000221+0000","last_became_active":"2024-03-27T09:34:48.981602+0000","last_became_peered":"2024-03-27T09:34:48.981602+0000","last_unstale":"2024-03-27T09:35:27.000221+0000","last_undegraded":"2024-03-27T09:35:27.000221+0000","last_fullsized":"2024-03-27T09:35:27.000221+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2201717,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":27298,"num_write":17,"num_write_kb":4161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":122,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.999025+0000","last_change":"2024-03-27T09:34:42.827902+0000","last_active":"2024-03-27T09:35:27.999025+0000","last_peered":"2024-03-27T09:35:27.999025+0000","last_clean":"2024-03-27T09:35:27.999025+0000","last_became_active":"2024-03-27T09:34:42.821715+0000","last_became_peered":"2024-03-27T09:34:42.821715+0000","last_unstale":"2024-03-27T09:35:27.999025+0000","last_undegraded":"2024-03-27T09:35:27.999025+0000","last_fullsized":"2024-03-27T09:35:27.999025+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"50'23","reported_seq":133,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.867564+0000","last_change":"2024-03-27T09:35:27.870122+0000","last_active":"2024-03-27T09:35:27.867564+0000","last_peered":"2024-03-27T09:35:27.867564+0000","last_clean":"2024-03-27T09:35:27.867564+0000","last_became_active":"2024-03-27T09:34:48.945926+0000","last_became_peered":"2024-03-27T09:34:48.945926+0000","last_unstale":"2024-03-27T09:35:27.867564+0000","last_undegraded":"2024-03-27T09:35:27.867564+0000","last_fullsized":"2024-03-27T09:35:27.867564+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8722767,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3051,"num_write":21,"num_write_kb":6018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"37'38","reported_seq":179,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.976102+0000","last_change":"2024-03-27T09:35:25.983399+0000","last_active":"2024-03-27T09:35:25.976102+0000","last_peered":"2024-03-27T09:35:25.976102+0000","last_clean":"2024-03-27T09:35:25.976102+0000","last_became_active":"2024-03-27T09:34:48.958003+0000","last_became_peered":"2024-03-27T09:34:48.958003+0000","last_unstale":"2024-03-27T09:35:25.976102+0000","last_undegraded":"2024-03-27T09:35:25.976102+0000","last_fullsized":"2024-03-27T09:35:25.976102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966452,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":32409,"num_write":44,"num_write_kb":10935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.765877+0000","last_change":"2024-03-27T09:34:42.824074+0000","last_active":"2024-03-27T09:35:25.765877+0000","last_peered":"2024-03-27T09:35:25.765877+0000","last_clean":"2024-03-27T09:35:25.765877+0000","last_became_active":"2024-03-27T09:34:42.819679+0000","last_became_peered":"2024-03-27T09:34:42.819679+0000","last_unstale":"2024-03-27T09:35:25.765877+0000","last_undegraded":"2024-03-27T09:35:25.765877+0000","last_fullsized":"2024-03-27T09:35:25.765877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"50'21","reported_seq":135,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.863395+0000","last_change":"2024-03-27T09:35:27.865894+0000","last_active":"2024-03-27T09:35:27.863395+0000","last_peered":"2024-03-27T09:35:27.863395+0000","last_clean":"2024-03-27T09:35:27.863395+0000","last_became_active":"2024-03-27T09:34:48.995086+0000","last_became_peered":"2024-03-27T09:34:48.995086+0000","last_unstale":"2024-03-27T09:35:27.863395+0000","last_undegraded":"2024-03-27T09:35:27.863395+0000","last_fullsized":"2024-03-27T09:35:27.863395+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4655506,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":6199,"num_write":21,"num_write_kb":5276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":105,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:26.999245+0000","last_change":"2024-03-27T09:34:42.825925+0000","last_active":"2024-03-27T09:35:26.999245+0000","last_peered":"2024-03-27T09:35:26.999245+0000","last_clean":"2024-03-27T09:35:26.999245+0000","last_became_active":"2024-03-27T09:34:42.820399+0000","last_became_peered":"2024-03-27T09:34:42.820399+0000","last_unstale":"2024-03-27T09:35:26.999245+0000","last_undegraded":"2024-03-27T09:35:26.999245+0000","last_fullsized":"2024-03-27T09:35:26.999245+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"50'79","reported_seq":200,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-27T09:35:27.004120+0000","last_change":"2024-03-27T09:35:27.008498+0000","last_active":"2024-03-27T09:35:27.004120+0000","last_peered":"2024-03-27T09:35:27.004120+0000","last_clean":"2024-03-27T09:35:27.004120+0000","last_became_active":"2024-03-27T09:34:49.013275+0000","last_became_peered":"2024-03-27T09:34:49.013275+0000","last_unstale":"2024-03-27T09:35:27.004120+0000","last_undegraded":"2024-03-27T09:35:27.004120+0000","last_fullsized":"2024-03-27T09:35:27.004120+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9327819,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":42,"num_read_kb":35527,"num_write":67,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.771855+0000","last_change":"2024-03-27T09:34:42.847708+0000","last_active":"2024-03-27T09:35:25.771855+0000","last_peered":"2024-03-27T09:35:25.771855+0000","last_clean":"2024-03-27T09:35:25.771855+0000","last_became_active":"2024-03-27T09:34:42.842086+0000","last_became_peered":"2024-03-27T09:34:42.842086+0000","last_unstale":"2024-03-27T09:35:25.771855+0000","last_undegraded":"2024-03-27T09:35:25.771855+0000","last_fullsized":"2024-03-27T09:35:25.771855+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":110,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.779486+0000","last_change":"2024-03-27T09:35:25.830878+0000","last_active":"2024-03-27T09:35:25.779486+0000","last_peered":"2024-03-27T09:35:25.779486+0000","last_clean":"2024-03-27T09:35:25.779486+0000","last_became_active":"2024-03-27T09:34:48.967426+0000","last_became_peered":"2024-03-27T09:34:48.967426+0000","last_unstale":"2024-03-27T09:35:25.779486+0000","last_undegraded":"2024-03-27T09:35:25.779486+0000","last_fullsized":"2024-03-27T09:35:25.779486+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:47.810415+0000","last_clean_scrub_stamp":"2024-03-27T09:34:47.810415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-27T09:35:25.983515+0000","last_change":"2024-03-27T09:34:42.840117+0000","last_active":"2024-03-27T09:35:25.983515+0000","last_peered":"2024-03-27T09:35:25.983515+0000","last_clean":"2024-03-27T09:35:25.983515+0000","last_became_active":"2024-03-27T09:34:42.835808+0000","last_became_peered":"2024-03-27T09:34:42.835808+0000","last_unstale":"2024-03-27T09:35:25.983515+0000","last_undegraded":"2024-03-27T09:35:25.983515+0000","last_fullsized":"2024-03-27T09:35:25.983515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T09:34:41.724047+0000","last_clean_scrub_stamp":"2024-03-27T09:34:41.724047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103614495,"num_objects":88,"num_object_clones":40,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":455,"num_read_kb":290397,"num_write":534,"num_write_kb":145572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":597,"ondisk_log_size":597,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67964,"kb_used_data":41140,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94303876,"statfs":{"total":96636764160,"available":96567169024,"internally_reserved":0,"allocated":42127360,"data_stored":41982416,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71832,"kb_used_data":45008,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300008,"statfs":{"total":96636764160,"available":96563208192,"internally_reserved":0,"allocated":46088192,"data_stored":46679381,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71804,"kb_used_data":44980,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300036,"statfs":{"total":96636764160,"available":96563236864,"internally_reserved":0,"allocated":46059520,"data_stored":46129117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67704,"kb_used_data":40880,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304136,"statfs":{"total":96636764160,"available":96567435264,"internally_reserved":0,"allocated":41861120,"data_stored":42514794,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:256: done (3 left) 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:257: done (2 left) 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:258: done (1 left) 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:259: done (0 left) 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:260: write (excl) oid 45 current snap is 21 2024-03-27T09:35:30.980 INFO:tasks.rados.rados.0.smithi170.stdout:260: seq_num 152 ranges {554664=448555,1478557=664373} 2024-03-27T09:35:30.987 INFO:tasks.rados.rados.0.smithi170.stdout:260: writing smithi17031214-45 from 554664 to 1003219 tid 1 2024-03-27T09:35:30.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: writing smithi17031214-45 from 1478557 to 2142930 tid 2 2024-03-27T09:35:30.996 INFO:tasks.rados.rados.0.smithi170.stdout:261: snap_remove snap 17 2024-03-27T09:35:31.984 INFO:tasks.rados.rados.0.smithi170.stdout:261: done (1 left) 2024-03-27T09:35:31.984 INFO:tasks.rados.rados.0.smithi170.stdout:262: write (excl) oid 9 current snap is 21 2024-03-27T09:35:31.984 INFO:tasks.rados.rados.0.smithi170.stdout:260: finishing write tid 1 to smithi17031214-45 2024-03-27T09:35:31.984 INFO:tasks.rados.rados.0.smithi170.stdout:262: seq_num 153 ranges {571471=640157,1653392=524384} 2024-03-27T09:35:31.990 INFO:tasks.rados.rados.0.smithi170.stdout:262: writing smithi17031214-9 from 571471 to 1211628 tid 1 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:262: writing smithi17031214-9 from 1653392 to 2177776 tid 2 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:263: write (excl) oid 20 current snap is 21 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: finishing write tid 2 to smithi17031214-45 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: finishing write tid 3 to smithi17031214-45 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: finishing write tid 4 to smithi17031214-45 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: oid 45 updating version 0 to 104 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: oid 45 updating version 104 to 106 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: oid 45 version 106 is already newer than 105 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 106 (ObjNum 152 snap 21 seq_num 152) dirty exists 2024-03-27T09:35:31.996 INFO:tasks.rados.rados.0.smithi170.stdout:260: left oid 45 (ObjNum 152 snap 21 seq_num 152) 2024-03-27T09:35:31.997 INFO:tasks.rados.rados.0.smithi170.stdout:263: seq_num 154 ranges {588278=715406,1711874=668042} 2024-03-27T09:35:32.006 INFO:tasks.rados.rados.0.smithi170.stdout:263: writing smithi17031214-20 from 588278 to 1303684 tid 1 2024-03-27T09:35:32.013 INFO:tasks.rados.rados.0.smithi170.stdout:263: writing smithi17031214-20 from 1711874 to 2379916 tid 2 2024-03-27T09:35:32.013 INFO:tasks.rados.rados.0.smithi170.stdout:260: done (2 left) 2024-03-27T09:35:32.013 INFO:tasks.rados.rados.0.smithi170.stdout:264: snap_create 2024-03-27T09:35:32.093 INFO:tasks.rados.rados.0.smithi170.stdout:262: finishing write tid 1 to smithi17031214-9 2024-03-27T09:35:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:263: finishing write tid 1 to smithi17031214-20 2024-03-27T09:35:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:262: finishing write tid 2 to smithi17031214-9 2024-03-27T09:35:32.132 INFO:tasks.rados.rados.0.smithi170.stdout:262: finishing write tid 3 to smithi17031214-9 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:262: finishing write tid 4 to smithi17031214-9 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:262: oid 9 updating version 0 to 78 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:262: oid 9 updating version 78 to 80 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:262: oid 9 version 80 is already newer than 80 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 80 (ObjNum 153 snap 21 seq_num 153) dirty exists 2024-03-27T09:35:32.134 INFO:tasks.rados.rados.0.smithi170.stdout:262: left oid 9 (ObjNum 153 snap 21 seq_num 153) 2024-03-27T09:35:32.152 INFO:tasks.rados.rados.0.smithi170.stdout:263: finishing write tid 2 to smithi17031214-20 2024-03-27T09:35:32.154 INFO:tasks.rados.rados.0.smithi170.stdout:263: finishing write tid 3 to smithi17031214-20 2024-03-27T09:35:32.156 INFO:tasks.rados.rados.0.smithi170.stdout:263: finishing write tid 4 to smithi17031214-20 2024-03-27T09:35:32.156 INFO:tasks.rados.rados.0.smithi170.stdout:263: oid 20 updating version 0 to 48 2024-03-27T09:35:32.156 INFO:tasks.rados.rados.0.smithi170.stdout:263: oid 20 version 48 is already newer than 48 2024-03-27T09:35:32.157 INFO:tasks.rados.rados.0.smithi170.stdout:263: oid 20 version 48 is already newer than 46 2024-03-27T09:35:32.157 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 48 (ObjNum 154 snap 21 seq_num 154) dirty exists 2024-03-27T09:35:32.157 INFO:tasks.rados.rados.0.smithi170.stdout:263: left oid 20 (ObjNum 154 snap 21 seq_num 154) 2024-03-27T09:35:32.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:32.587 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:32.584+0000 7f489675d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:32.591 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:32.589+0000 7f489675d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:32.986 INFO:tasks.rados.rados.0.smithi170.stdout:262: done (2 left) 2024-03-27T09:35:32.987 INFO:tasks.rados.rados.0.smithi170.stdout:263: done (1 left) 2024-03-27T09:35:32.987 INFO:tasks.rados.rados.0.smithi170.stdout:264: done (0 left) 2024-03-27T09:35:32.987 INFO:tasks.rados.rados.0.smithi170.stdout:265: snap_create 2024-03-27T09:35:33.991 INFO:tasks.rados.rados.0.smithi170.stdout:265: done (0 left) 2024-03-27T09:35:33.991 INFO:tasks.rados.rados.0.smithi170.stdout:266: snap_remove snap 22 2024-03-27T09:35:34.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:35:34.815 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:34.813+0000 7f28858ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:34.819 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:34.818+0000 7f28858ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:34.992 INFO:tasks.rados.rados.0.smithi170.stdout:266: done (0 left) 2024-03-27T09:35:34.992 INFO:tasks.rados.rados.0.smithi170.stdout:267: write (excl) oid 9 current snap is 23 2024-03-27T09:35:34.992 INFO:tasks.rados.rados.0.smithi170.stdout:267: seq_num 155 ranges {605085=790655,2170356=434729} 2024-03-27T09:35:34.999 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17031214-9 from 605085 to 1395740 tid 1 2024-03-27T09:35:35.006 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17031214-9 from 2170356 to 2605085 tid 2 2024-03-27T09:35:35.006 INFO:tasks.rados.rados.0.smithi170.stdout:268: snap_create 2024-03-27T09:35:35.090 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 1 to smithi17031214-9 2024-03-27T09:35:35.112 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 2 to smithi17031214-9 2024-03-27T09:35:35.116 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 3 to smithi17031214-9 2024-03-27T09:35:35.117 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 4 to smithi17031214-9 2024-03-27T09:35:35.117 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 9 updating version 0 to 84 2024-03-27T09:35:35.117 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 9 version 84 is already newer than 84 2024-03-27T09:35:35.118 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 9 version 84 is already newer than 82 2024-03-27T09:35:35.118 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 84 (ObjNum 155 snap 23 seq_num 155) dirty exists 2024-03-27T09:35:35.118 INFO:tasks.rados.rados.0.smithi170.stdout:267: left oid 9 (ObjNum 155 snap 23 seq_num 155) 2024-03-27T09:35:35.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-27T09:35:35.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:35.866 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2024-03-27T09:35:35.866 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T09:35:35.867 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T09:35:35.933 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T09:35:35.997 INFO:tasks.rados.rados.0.smithi170.stdout:267: done (1 left) 2024-03-27T09:35:35.997 INFO:tasks.rados.rados.0.smithi170.stdout:268: done (0 left) 2024-03-27T09:35:35.997 INFO:tasks.rados.rados.0.smithi170.stdout:269: read oid 33 snap -1 2024-03-27T09:35:35.997 INFO:tasks.rados.rados.0.smithi170.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-27T09:35:35.997 INFO:tasks.rados.rados.0.smithi170.stdout:270: delete oid 39 current snap is 24 2024-03-27T09:35:36.045 INFO:tasks.rados.rados.0.smithi170.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-27T09:35:36.154 INFO:tasks.rados.rados.0.smithi170.stdout:269: done (1 left) 2024-03-27T09:35:36.154 INFO:tasks.rados.rados.0.smithi170.stdout:270: done (0 left) 2024-03-27T09:35:36.154 INFO:tasks.rados.rados.0.smithi170.stdout:271: snap_remove snap 21 2024-03-27T09:35:36.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:35:36.999 INFO:tasks.rados.rados.0.smithi170.stdout:271: done (0 left) 2024-03-27T09:35:36.999 INFO:tasks.rados.rados.0.smithi170.stdout:272: write (excl) oid 2 current snap is 24 2024-03-27T09:35:36.999 INFO:tasks.rados.rados.0.smithi170.stdout:272: seq_num 157 ranges {638699=541153,1770967=531722} 2024-03-27T09:35:37.005 INFO:tasks.rados.rados.0.smithi170.stdout:272: writing smithi17031214-2 from 638699 to 1179852 tid 1 2024-03-27T09:35:37.012 INFO:tasks.rados.rados.0.smithi170.stdout:272: writing smithi17031214-2 from 1770967 to 2302689 tid 2 2024-03-27T09:35:37.012 INFO:tasks.rados.rados.0.smithi170.stdout:273: delete oid 8 current snap is 24 2024-03-27T09:35:37.072 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:37.069+0000 7fbbc026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:37.076 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:35:37.075+0000 7fbbc026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:35:37.083 INFO:tasks.rados.rados.0.smithi170.stdout:273: done (1 left) 2024-03-27T09:35:37.083 INFO:tasks.rados.rados.0.smithi170.stdout:274: delete oid 42 current snap is 24 2024-03-27T09:35:37.126 INFO:tasks.rados.rados.0.smithi170.stdout:274: done (1 left) 2024-03-27T09:35:37.126 INFO:tasks.rados.rados.0.smithi170.stdout:275: read oid 27 snap -1 2024-03-27T09:35:37.126 INFO:tasks.rados.rados.0.smithi170.stdout:275: expect (ObjNum 99 snap 8 seq_num 99) 2024-03-27T09:35:37.126 INFO:tasks.rados.rados.0.smithi170.stdout:276: snap_remove snap 20 2024-03-27T09:35:38.006 INFO:tasks.rados.rados.0.smithi170.stdout:276: done (2 left) 2024-03-27T09:35:38.006 INFO:tasks.rados.rados.0.smithi170.stdout:277: delete oid 32 current snap is 24 2024-03-27T09:35:40.933 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2024-03-27T09:35:40.933 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-27T09:35:40.933 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-27T09:35:40.934 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T09:35:40.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-27T09:35:40.936 INFO:tasks.ceph.osd.1:Started 2024-03-27T09:35:40.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:35:41.045 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:35:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:41.049 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:35:46.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:35:46.153 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:35:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:46.159 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:35:50.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-27T09:35:51.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:35:51.275 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:35:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:51.278 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:35:55.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2024-03-27T09:35:56.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:35:56.390 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:35:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:35:56.393 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:01.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:01.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-03-27T09:36:01.506 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:01.509 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:02.021 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:02.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:03.114 INFO:tasks.rados.rados.0.smithi170.stdout:277: done (2 left) 2024-03-27T09:36:03.115 INFO:tasks.rados.rados.0.smithi170.stdout:278: write (excl) oid 39 current snap is 24 2024-03-27T09:36:03.115 INFO:tasks.rados.rados.0.smithi170.stdout:278: seq_num 161 ranges {705927=558502,2004895=422707} 2024-03-27T09:36:03.121 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17031214-39 from 705927 to 1264429 tid 1 2024-03-27T09:36:03.128 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17031214-39 from 2004895 to 2427602 tid 2 2024-03-27T09:36:03.128 INFO:tasks.rados.rados.0.smithi170.stdout:279: write (excl) oid 36 current snap is 24 2024-03-27T09:36:03.128 INFO:tasks.rados.rados.0.smithi170.stdout:279: seq_num 162 ranges {722734=633751,2063377=659357} 2024-03-27T09:36:03.136 INFO:tasks.rados.rados.0.smithi170.stdout:279: writing smithi17031214-36 from 722734 to 1356485 tid 1 2024-03-27T09:36:03.144 INFO:tasks.rados.rados.0.smithi170.stdout:279: writing smithi17031214-36 from 2063377 to 2722734 tid 2 2024-03-27T09:36:03.144 INFO:tasks.rados.rados.0.smithi170.stdout:280: snap_remove snap 18 2024-03-27T09:36:04.030 INFO:tasks.rados.rados.0.smithi170.stdout:280: done (4 left) 2024-03-27T09:36:04.030 INFO:tasks.rados.rados.0.smithi170.stdout:281: write (excl) oid 17 current snap is 24 2024-03-27T09:36:04.031 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 1 to smithi17031214-39 2024-03-27T09:36:04.031 INFO:tasks.rados.rados.0.smithi170.stdout:281: seq_num 163 ranges {739541=709000,2121859=426376} 2024-03-27T09:36:04.038 INFO:tasks.rados.rados.0.smithi170.stdout:281: writing smithi17031214-17 from 739541 to 1448541 tid 1 2024-03-27T09:36:04.044 INFO:tasks.rados.rados.0.smithi170.stdout:281: writing smithi17031214-17 from 2121859 to 2548235 tid 2 2024-03-27T09:36:04.044 INFO:tasks.rados.rados.0.smithi170.stdout:282: write oid 25 current snap is 24 2024-03-27T09:36:04.044 INFO:tasks.rados.rados.0.smithi170.stdout:282: seq_num 164 ranges {756348=784249,2063988=686387} 2024-03-27T09:36:04.052 INFO:tasks.rados.rados.0.smithi170.stdout:282: writing smithi17031214-25 from 756348 to 1540597 tid 1 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:282: writing smithi17031214-25 from 2063988 to 2750375 tid 2 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:283: snap_create 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing write tid 1 to smithi17031214-36 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing write tid 1 to smithi17031214-2 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing write tid 2 to smithi17031214-2 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 2 to smithi17031214-39 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing write tid 3 to smithi17031214-2 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing write tid 2 to smithi17031214-36 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing write tid 4 to smithi17031214-2 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: oid 2 updating version 0 to 112 2024-03-27T09:36:04.060 INFO:tasks.rados.rados.0.smithi170.stdout:272: oid 2 version 112 is already newer than 110 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:272: oid 2 version 112 is already newer than 112 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 112 (ObjNum 157 snap 24 seq_num 157) dirty exists 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:272: left oid 2 (ObjNum 157 snap 24 seq_num 157) 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing write tid 3 to smithi17031214-36 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing write tid 4 to smithi17031214-36 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: oid 36 updating version 0 to 83 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: oid 36 version 83 is already newer than 82 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: oid 36 updating version 83 to 84 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 84 (ObjNum 162 snap 24 seq_num 162) dirty exists 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:279: left oid 36 (ObjNum 162 snap 24 seq_num 162) 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 3 to smithi17031214-39 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 4 to smithi17031214-39 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 39 updating version 0 to 31 2024-03-27T09:36:04.061 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 39 updating version 31 to 32 2024-03-27T09:36:04.062 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 39 version 32 is already newer than 30 2024-03-27T09:36:04.062 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 32 (ObjNum 161 snap 24 seq_num 161) dirty exists 2024-03-27T09:36:04.062 INFO:tasks.rados.rados.0.smithi170.stdout:278: left oid 39 (ObjNum 161 snap 24 seq_num 161) 2024-03-27T09:36:04.062 INFO:tasks.rados.rados.0.smithi170.stdout:275: expect (ObjNum 99 snap 8 seq_num 99) 2024-03-27T09:36:04.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:04.282 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:04.280+0000 7f1bfd9ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:04.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:04.285+0000 7f1bfd9ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:281: finishing write tid 1 to smithi17031214-17 2024-03-27T09:36:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:282: finishing write tid 1 to smithi17031214-25 2024-03-27T09:36:04.293 INFO:tasks.rados.rados.0.smithi170.stdout:281: finishing write tid 2 to smithi17031214-17 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:282: finishing write tid 2 to smithi17031214-25 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: finishing write tid 3 to smithi17031214-17 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:282: finishing write tid 3 to smithi17031214-25 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: finishing write tid 4 to smithi17031214-17 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: oid 17 updating version 0 to 87 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: oid 17 version 87 is already newer than 85 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: oid 17 version 87 is already newer than 86 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 87 (ObjNum 163 snap 24 seq_num 163) dirty exists 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:281: left oid 17 (ObjNum 163 snap 24 seq_num 163) 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:282: finishing write tid 4 to smithi17031214-25 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:282: oid 25 updating version 0 to 114 2024-03-27T09:36:04.294 INFO:tasks.rados.rados.0.smithi170.stdout:282: oid 25 updating version 114 to 115 2024-03-27T09:36:04.295 INFO:tasks.rados.rados.0.smithi170.stdout:282: oid 25 updating version 115 to 116 2024-03-27T09:36:04.295 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 116 (ObjNum 164 snap 24 seq_num 164) dirty exists 2024-03-27T09:36:04.295 INFO:tasks.rados.rados.0.smithi170.stdout:282: left oid 25 (ObjNum 164 snap 24 seq_num 164) 2024-03-27T09:36:04.309 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:04.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:05.034 INFO:tasks.rados.rados.0.smithi170.stdout:272: done (6 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:275: done (5 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:278: done (4 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:279: done (3 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:281: done (2 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:282: done (1 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:283: done (0 left) 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:284: read oid 13 snap -1 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:284: expect deleted 2024-03-27T09:36:05.035 INFO:tasks.rados.rados.0.smithi170.stdout:285: snap_remove snap 24 2024-03-27T09:36:05.941 INFO:tasks.rados.rados.0.smithi170.stdout:285: done (1 left) 2024-03-27T09:36:05.941 INFO:tasks.rados.rados.0.smithi170.stdout:286: write oid 9 current snap is 25 2024-03-27T09:36:05.942 INFO:tasks.rados.rados.0.smithi170.stdout:286: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-03-27T09:36:05.947 INFO:tasks.rados.rados.0.smithi170.stdout:286: writing smithi17031214-9 from 773155 to 1232653 tid 1 2024-03-27T09:36:05.953 INFO:tasks.rados.rados.0.smithi170.stdout:286: writing smithi17031214-9 from 1722470 to 2152515 tid 2 2024-03-27T09:36:05.956 INFO:tasks.rados.rados.0.smithi170.stdout:286: writing smithi17031214-9 from 2648676 to 2773155 tid 3 2024-03-27T09:36:05.956 INFO:tasks.rados.rados.0.smithi170.stdout:287: delete oid 26 current snap is 25 2024-03-27T09:36:06.048 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing write tid 1 to smithi17031214-9 2024-03-27T09:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:284: done (2 left) 2024-03-27T09:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:287: done (1 left) 2024-03-27T09:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:288: read oid 6 snap 23 2024-03-27T09:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:288: expect deleted 2024-03-27T09:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:289: snap_create 2024-03-27T09:36:06.081 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing write tid 2 to smithi17031214-9 2024-03-27T09:36:06.101 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing write tid 3 to smithi17031214-9 2024-03-27T09:36:06.109 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing write tid 4 to smithi17031214-9 2024-03-27T09:36:06.110 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing write tid 5 to smithi17031214-9 2024-03-27T09:36:06.110 INFO:tasks.rados.rados.0.smithi170.stdout:286: oid 9 updating version 0 to 101 2024-03-27T09:36:06.110 INFO:tasks.rados.rados.0.smithi170.stdout:286: oid 9 version 101 is already newer than 98 2024-03-27T09:36:06.110 INFO:tasks.rados.rados.0.smithi170.stdout:286: oid 9 version 101 is already newer than 100 2024-03-27T09:36:06.111 INFO:tasks.rados.rados.0.smithi170.stdout:286: oid 9 version 101 is already newer than 99 2024-03-27T09:36:06.111 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 101 (ObjNum 165 snap 25 seq_num 165) dirty exists 2024-03-27T09:36:06.111 INFO:tasks.rados.rados.0.smithi170.stdout:286: left oid 9 (ObjNum 165 snap 25 seq_num 165) 2024-03-27T09:36:06.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:06.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:06.536 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:06.534+0000 7fb5f959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:06.540 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:06.539+0000 7fb5f959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:06.565 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:06.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:06.614 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:06.617 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:06.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2024-03-27T09:36:07.040 INFO:tasks.rados.rados.0.smithi170.stdout:286: done (2 left) 2024-03-27T09:36:07.040 INFO:tasks.rados.rados.0.smithi170.stdout:288: done (1 left) 2024-03-27T09:36:07.040 INFO:tasks.rados.rados.0.smithi170.stdout:289: done (0 left) 2024-03-27T09:36:07.040 INFO:tasks.rados.rados.0.smithi170.stdout:290: write (excl) oid 7 current snap is 26 2024-03-27T09:36:07.040 INFO:tasks.rados.rados.0.smithi170.stdout:290: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-03-27T09:36:07.045 INFO:tasks.rados.rados.0.smithi170.stdout:290: writing smithi17031214-7 from 406769 to 1016765 tid 1 2024-03-27T09:36:07.050 INFO:tasks.rados.rados.0.smithi170.stdout:290: writing smithi17031214-7 from 1439434 to 1873148 tid 2 2024-03-27T09:36:07.055 INFO:tasks.rados.rados.0.smithi170.stdout:290: writing smithi17031214-7 from 2303522 to 2806769 tid 3 2024-03-27T09:36:07.055 INFO:tasks.rados.rados.0.smithi170.stdout:291: read oid 50 snap -1 2024-03-27T09:36:07.055 INFO:tasks.rados.rados.0.smithi170.stdout:291: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-27T09:36:07.055 INFO:tasks.rados.rados.0.smithi170.stdout:292: read oid 36 snap -1 2024-03-27T09:36:07.055 INFO:tasks.rados.rados.0.smithi170.stdout:292: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-27T09:36:07.056 INFO:tasks.rados.rados.0.smithi170.stdout:293: read oid 20 snap 23 2024-03-27T09:36:07.056 INFO:tasks.rados.rados.0.smithi170.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2024-03-27T09:36:07.056 INFO:tasks.rados.rados.0.smithi170.stdout:294: delete oid 23 current snap is 26 2024-03-27T09:36:07.095 INFO:tasks.rados.rados.0.smithi170.stdout:294: done (4 left) 2024-03-27T09:36:07.095 INFO:tasks.rados.rados.0.smithi170.stdout:295: write oid 3 current snap is 26 2024-03-27T09:36:07.096 INFO:tasks.rados.rados.0.smithi170.stdout:295: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-03-27T09:36:07.100 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17031214-3 from 440383 to 917230 tid 1 2024-03-27T09:36:07.108 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17031214-3 from 1556398 to 1993781 tid 2 2024-03-27T09:36:07.113 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17031214-3 from 2474721 to 2840383 tid 3 2024-03-27T09:36:07.114 INFO:tasks.rados.rados.0.smithi170.stdout:296: write oid 9 current snap is 26 2024-03-27T09:36:07.114 INFO:tasks.rados.rados.0.smithi170.stdout:296: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-03-27T09:36:07.119 INFO:tasks.rados.rados.0.smithi170.stdout:296: writing smithi17031214-9 from 457190 to 1009286 tid 1 2024-03-27T09:36:07.126 INFO:tasks.rados.rados.0.smithi170.stdout:296: writing smithi17031214-9 from 1614880 to 2195921 tid 2 2024-03-27T09:36:07.130 INFO:tasks.rados.rados.0.smithi170.stdout:296: writing smithi17031214-9 from 2785791 to 2857190 tid 3 2024-03-27T09:36:07.130 INFO:tasks.rados.rados.0.smithi170.stdout:297: snap_create 2024-03-27T09:36:07.143 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 1 to smithi17031214-3 2024-03-27T09:36:07.152 INFO:tasks.rados.rados.0.smithi170.stdout:291: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-27T09:36:07.325 INFO:tasks.rados.rados.0.smithi170.stdout:292: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing write tid 1 to smithi17031214-7 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 2 to smithi17031214-3 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 3 to smithi17031214-3 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 4 to smithi17031214-3 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 5 to smithi17031214-3 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 3 updating version 0 to 51 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 3 updating version 51 to 54 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 3 version 54 is already newer than 54 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 3 version 54 is already newer than 52 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 54 (ObjNum 169 snap 26 seq_num 169) dirty exists 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:295: left oid 3 (ObjNum 169 snap 26 seq_num 169) 2024-03-27T09:36:07.456 INFO:tasks.rados.rados.0.smithi170.stdout:293: expect (ObjNum 154 snap 21 seq_num 154) 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing write tid 2 to smithi17031214-7 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:296: finishing write tid 1 to smithi17031214-9 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing write tid 3 to smithi17031214-7 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:296: finishing write tid 2 to smithi17031214-9 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing write tid 4 to smithi17031214-7 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing write tid 5 to smithi17031214-7 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: oid 7 updating version 0 to 30 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: oid 7 updating version 30 to 33 2024-03-27T09:36:07.608 INFO:tasks.rados.rados.0.smithi170.stdout:290: oid 7 version 33 is already newer than 32 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:290: oid 7 version 33 is already newer than 31 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 33 (ObjNum 167 snap 26 seq_num 167) dirty exists 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:290: left oid 7 (ObjNum 167 snap 26 seq_num 167) 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: finishing write tid 3 to smithi17031214-9 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: finishing write tid 4 to smithi17031214-9 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: finishing write tid 5 to smithi17031214-9 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: oid 9 updating version 0 to 105 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: oid 9 updating version 105 to 107 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: oid 9 version 107 is already newer than 106 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: oid 9 version 107 is already newer than 104 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 107 (ObjNum 170 snap 26 seq_num 170) dirty exists 2024-03-27T09:36:07.609 INFO:tasks.rados.rados.0.smithi170.stdout:296: left oid 9 (ObjNum 170 snap 26 seq_num 170) 2024-03-27T09:36:08.043 INFO:tasks.rados.rados.0.smithi170.stdout:290: done (6 left) 2024-03-27T09:36:08.043 INFO:tasks.rados.rados.0.smithi170.stdout:291: done (5 left) 2024-03-27T09:36:08.043 INFO:tasks.rados.rados.0.smithi170.stdout:292: done (4 left) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:293: done (3 left) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:295: done (2 left) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:296: done (1 left) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:297: done (0 left) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:298: read oid 24 snap 25 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:298: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-27T09:36:08.044 INFO:tasks.rados.rados.0.smithi170.stdout:299: snap_remove snap 23 2024-03-27T09:36:08.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:08.725 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:08.723+0000 7fed3419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:08.729 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:08.728+0000 7fed3419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:08.753 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:08.803 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:09.046 INFO:tasks.rados.rados.0.smithi170.stdout:299: done (1 left) 2024-03-27T09:36:09.046 INFO:tasks.rados.rados.0.smithi170.stdout:300: read oid 6 snap -1 2024-03-27T09:36:09.046 INFO:tasks.rados.rados.0.smithi170.stdout:300: expect deleted 2024-03-27T09:36:09.046 INFO:tasks.rados.rados.0.smithi170.stdout:301: snap_remove snap 26 2024-03-27T09:36:10.052 INFO:tasks.rados.rados.0.smithi170.stdout:301: done (2 left) 2024-03-27T09:36:10.052 INFO:tasks.rados.rados.0.smithi170.stdout:302: write oid 31 current snap is 27 2024-03-27T09:36:10.052 INFO:tasks.rados.rados.0.smithi170.stdout:302: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-03-27T09:36:10.057 INFO:tasks.rados.rados.0.smithi170.stdout:302: writing smithi17031214-31 from 473997 to 1101342 tid 1 2024-03-27T09:36:10.063 INFO:tasks.rados.rados.0.smithi170.stdout:302: writing smithi17031214-31 from 1673362 to 2114414 tid 2 2024-03-27T09:36:10.068 INFO:tasks.rados.rados.0.smithi170.stdout:302: writing smithi17031214-31 from 2529567 to 2873997 tid 3 2024-03-27T09:36:10.068 INFO:tasks.rados.rados.0.smithi170.stdout:303: read oid 39 snap -1 2024-03-27T09:36:10.068 INFO:tasks.rados.rados.0.smithi170.stdout:303: expect (ObjNum 161 snap 24 seq_num 161) 2024-03-27T09:36:10.068 INFO:tasks.rados.rados.0.smithi170.stdout:304: write (excl) oid 17 current snap is 27 2024-03-27T09:36:10.068 INFO:tasks.rados.rados.0.smithi170.stdout:304: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-03-27T09:36:10.075 INFO:tasks.rados.rados.0.smithi170.stdout:304: writing smithi17031214-17 from 490804 to 1193398 tid 1 2024-03-27T09:36:10.081 INFO:tasks.rados.rados.0.smithi170.stdout:304: writing smithi17031214-17 from 1615491 to 2200201 tid 2 2024-03-27T09:36:10.085 INFO:tasks.rados.rados.0.smithi170.stdout:304: writing smithi17031214-17 from 2724284 to 2890804 tid 3 2024-03-27T09:36:10.085 INFO:tasks.rados.rados.0.smithi170.stdout:305: snap_create 2024-03-27T09:36:10.086 INFO:tasks.rados.rados.0.smithi170.stdout:298: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-27T09:36:10.181 INFO:tasks.rados.rados.0.smithi170.stdout:303: expect (ObjNum 161 snap 24 seq_num 161) 2024-03-27T09:36:10.286 INFO:tasks.rados.rados.0.smithi170.stdout:302: finishing write tid 1 to smithi17031214-31 2024-03-27T09:36:10.286 INFO:tasks.rados.rados.0.smithi170.stdout:304: finishing write tid 1 to smithi17031214-17 2024-03-27T09:36:10.286 INFO:tasks.rados.rados.0.smithi170.stdout:302: finishing write tid 2 to smithi17031214-31 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: finishing write tid 2 to smithi17031214-17 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:302: finishing write tid 3 to smithi17031214-31 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: finishing write tid 3 to smithi17031214-17 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: finishing write tid 4 to smithi17031214-17 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: finishing write tid 5 to smithi17031214-17 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: oid 17 updating version 0 to 93 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: oid 17 updating version 93 to 96 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: oid 17 version 96 is already newer than 94 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: oid 17 version 96 is already newer than 96 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 96 (ObjNum 172 snap 27 seq_num 172) dirty exists 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:304: left oid 17 (ObjNum 172 snap 27 seq_num 172) 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:302: finishing write tid 4 to smithi17031214-31 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:302: finishing write tid 5 to smithi17031214-31 2024-03-27T09:36:10.287 INFO:tasks.rados.rados.0.smithi170.stdout:302: oid 31 updating version 0 to 56 2024-03-27T09:36:10.288 INFO:tasks.rados.rados.0.smithi170.stdout:302: oid 31 updating version 56 to 57 2024-03-27T09:36:10.288 INFO:tasks.rados.rados.0.smithi170.stdout:302: oid 31 version 57 is already newer than 54 2024-03-27T09:36:10.288 INFO:tasks.rados.rados.0.smithi170.stdout:302: oid 31 version 57 is already newer than 55 2024-03-27T09:36:10.288 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 57 (ObjNum 171 snap 27 seq_num 171) dirty exists 2024-03-27T09:36:10.288 INFO:tasks.rados.rados.0.smithi170.stdout:302: left oid 31 (ObjNum 171 snap 27 seq_num 171) 2024-03-27T09:36:10.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:10.915 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:10.913+0000 7fdeb219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:10.919 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:10.918+0000 7fdeb219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:10.943 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:10.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:298: done (5 left) 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:300: done (4 left) 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:302: done (3 left) 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:303: done (2 left) 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:304: done (1 left) 2024-03-27T09:36:11.054 INFO:tasks.rados.rados.0.smithi170.stdout:305: done (0 left) 2024-03-27T09:36:11.055 INFO:tasks.rados.rados.0.smithi170.stdout:306: read oid 16 snap -1 2024-03-27T09:36:11.055 INFO:tasks.rados.rados.0.smithi170.stdout:306: expect deleted 2024-03-27T09:36:11.055 INFO:tasks.rados.rados.0.smithi170.stdout:307: delete oid 37 current snap is 28 2024-03-27T09:36:11.114 INFO:tasks.rados.rados.0.smithi170.stdout:306: done (1 left) 2024-03-27T09:36:11.114 INFO:tasks.rados.rados.0.smithi170.stdout:307: done (0 left) 2024-03-27T09:36:11.115 INFO:tasks.rados.rados.0.smithi170.stdout:308: snap_remove snap 27 2024-03-27T09:36:11.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:11.717 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:11.720 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:11.720 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:12.057 INFO:tasks.rados.rados.0.smithi170.stdout:308: done (0 left) 2024-03-27T09:36:12.057 INFO:tasks.rados.rados.0.smithi170.stdout:309: snap_create 2024-03-27T09:36:12.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-03-27T09:36:12.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:13.061 INFO:tasks.rados.rados.0.smithi170.stdout:309: done (0 left) 2024-03-27T09:36:13.061 INFO:tasks.rados.rados.0.smithi170.stdout:310: write oid 32 current snap is 29 2024-03-27T09:36:13.061 INFO:tasks.rados.rados.0.smithi170.stdout:310: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-03-27T09:36:13.068 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17031214-32 from 524418 to 977510 tid 1 2024-03-27T09:36:13.076 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17031214-32 from 1732455 to 2320834 tid 2 2024-03-27T09:36:13.080 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17031214-32 from 2779130 to 2924418 tid 3 2024-03-27T09:36:13.080 INFO:tasks.rados.rados.0.smithi170.stdout:311: read oid 36 snap 28 2024-03-27T09:36:13.080 INFO:tasks.rados.rados.0.smithi170.stdout:311: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-27T09:36:13.080 INFO:tasks.rados.rados.0.smithi170.stdout:312: snap_create 2024-03-27T09:36:13.132 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:13.130+0000 7fe41cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:13.137 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:13.136+0000 7fe41cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:13.167 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:13.176 INFO:tasks.rados.rados.0.smithi170.stdout:311: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-27T09:36:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:13.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 1 to smithi17031214-32 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 2 to smithi17031214-32 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 3 to smithi17031214-32 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 4 to smithi17031214-32 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 5 to smithi17031214-32 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 32 updating version 0 to 84 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 32 version 84 is already newer than 83 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 32 version 84 is already newer than 81 2024-03-27T09:36:13.312 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 32 version 84 is already newer than 82 2024-03-27T09:36:13.313 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 84 (ObjNum 174 snap 29 seq_num 174) dirty exists 2024-03-27T09:36:13.313 INFO:tasks.rados.rados.0.smithi170.stdout:310: left oid 32 (ObjNum 174 snap 29 seq_num 174) 2024-03-27T09:36:14.064 INFO:tasks.rados.rados.0.smithi170.stdout:310: done (2 left) 2024-03-27T09:36:14.065 INFO:tasks.rados.rados.0.smithi170.stdout:311: done (1 left) 2024-03-27T09:36:14.065 INFO:tasks.rados.rados.0.smithi170.stdout:312: done (0 left) 2024-03-27T09:36:14.065 INFO:tasks.rados.rados.0.smithi170.stdout:313: read oid 23 snap -1 2024-03-27T09:36:14.065 INFO:tasks.rados.rados.0.smithi170.stdout:313: expect deleted 2024-03-27T09:36:14.065 INFO:tasks.rados.rados.0.smithi170.stdout:314: snap_create 2024-03-27T09:36:15.066 INFO:tasks.rados.rados.0.smithi170.stdout:313: done (1 left) 2024-03-27T09:36:15.066 INFO:tasks.rados.rados.0.smithi170.stdout:314: done (0 left) 2024-03-27T09:36:15.066 INFO:tasks.rados.rados.0.smithi170.stdout:315: snap_remove snap 30 2024-03-27T09:36:15.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:15.343 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:15.340+0000 7f0af7a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:15.346 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:15.345+0000 7f0af7a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:15.370 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:15.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:15.943 INFO:tasks.rados.rados.0.smithi170.stdout:315: done (0 left) 2024-03-27T09:36:15.950 INFO:tasks.rados.rados.0.smithi170.stdout:316: snap_create 2024-03-27T09:36:16.721 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:16.832 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:16.835 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:16.943 INFO:tasks.rados.rados.0.smithi170.stdout:316: done (0 left) 2024-03-27T09:36:16.943 INFO:tasks.rados.rados.0.smithi170.stdout:317: read oid 34 snap -1 2024-03-27T09:36:16.944 INFO:tasks.rados.rados.0.smithi170.stdout:317: expect (ObjNum 33 snap 0 seq_num 33) 2024-03-27T09:36:16.944 INFO:tasks.rados.rados.0.smithi170.stdout:318: delete oid 2 current snap is 32 2024-03-27T09:36:17.026 INFO:tasks.rados.rados.0.smithi170.stdout:317: expect (ObjNum 33 snap 0 seq_num 33) 2024-03-27T09:36:17.127 INFO:tasks.rados.rados.0.smithi170.stdout:317: done (1 left) 2024-03-27T09:36:17.127 INFO:tasks.rados.rados.0.smithi170.stdout:318: done (0 left) 2024-03-27T09:36:17.127 INFO:tasks.rados.rados.0.smithi170.stdout:319: snap_create 2024-03-27T09:36:17.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:17.531 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:17.529+0000 7f8bdbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:17.534+0000 7f8bdbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:17.558 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:17.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:17.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2024-03-27T09:36:18.074 INFO:tasks.rados.rados.0.smithi170.stdout:319: done (0 left) 2024-03-27T09:36:18.075 INFO:tasks.rados.rados.0.smithi170.stdout:320: read oid 2 snap -1 2024-03-27T09:36:18.075 INFO:tasks.rados.rados.0.smithi170.stdout:320: expect deleted 2024-03-27T09:36:18.075 INFO:tasks.rados.rados.0.smithi170.stdout:321: write (excl) oid 16 current snap is 33 2024-03-27T09:36:18.075 INFO:tasks.rados.rados.0.smithi170.stdout:321: seq_num 176 ranges {558032=719943,1849419=592048} 2024-03-27T09:36:18.081 INFO:tasks.rados.rados.0.smithi170.stdout:321: writing smithi17031214-16 from 558032 to 1277975 tid 1 2024-03-27T09:36:18.087 INFO:tasks.rados.rados.0.smithi170.stdout:321: writing smithi17031214-16 from 1849419 to 2441467 tid 2 2024-03-27T09:36:18.087 INFO:tasks.rados.rados.0.smithi170.stdout:322: write oid 13 current snap is 33 2024-03-27T09:36:18.088 INFO:tasks.rados.rados.0.smithi170.stdout:322: seq_num 177 ranges {574839=795192,1907901=452059} 2024-03-27T09:36:18.095 INFO:tasks.rados.rados.0.smithi170.stdout:322: writing smithi17031214-13 from 574839 to 1370031 tid 1 2024-03-27T09:36:18.101 INFO:tasks.rados.rados.0.smithi170.stdout:322: writing smithi17031214-13 from 1907901 to 2359960 tid 2 2024-03-27T09:36:18.101 INFO:tasks.rados.rados.0.smithi170.stdout:323: delete oid 22 current snap is 33 2024-03-27T09:36:18.163 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing write tid 1 to smithi17031214-16 2024-03-27T09:36:18.179 INFO:tasks.rados.rados.0.smithi170.stdout:322: finishing write tid 1 to smithi17031214-13 2024-03-27T09:36:18.190 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing write tid 2 to smithi17031214-16 2024-03-27T09:36:18.196 INFO:tasks.rados.rados.0.smithi170.stdout:320: done (3 left) 2024-03-27T09:36:18.196 INFO:tasks.rados.rados.0.smithi170.stdout:323: done (2 left) 2024-03-27T09:36:18.197 INFO:tasks.rados.rados.0.smithi170.stdout:324: delete oid 33 current snap is 33 2024-03-27T09:36:18.202 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing write tid 3 to smithi17031214-16 2024-03-27T09:36:18.203 INFO:tasks.rados.rados.0.smithi170.stdout:322: finishing write tid 2 to smithi17031214-13 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing write tid 4 to smithi17031214-16 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:321: oid 16 updating version 0 to 37 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:321: oid 16 version 37 is already newer than 36 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:321: oid 16 updating version 37 to 38 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 38 (ObjNum 176 snap 33 seq_num 176) dirty exists 2024-03-27T09:36:18.206 INFO:tasks.rados.rados.0.smithi170.stdout:321: left oid 16 (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:36:18.210 INFO:tasks.rados.rados.0.smithi170.stdout:322: finishing write tid 3 to smithi17031214-13 2024-03-27T09:36:18.213 INFO:tasks.rados.rados.0.smithi170.stdout:322: finishing write tid 4 to smithi17031214-13 2024-03-27T09:36:18.213 INFO:tasks.rados.rados.0.smithi170.stdout:322: oid 13 updating version 0 to 39 2024-03-27T09:36:18.214 INFO:tasks.rados.rados.0.smithi170.stdout:322: oid 13 version 39 is already newer than 37 2024-03-27T09:36:18.214 INFO:tasks.rados.rados.0.smithi170.stdout:322: oid 13 version 39 is already newer than 39 2024-03-27T09:36:18.214 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 39 (ObjNum 177 snap 33 seq_num 177) dirty exists 2024-03-27T09:36:18.214 INFO:tasks.rados.rados.0.smithi170.stdout:322: left oid 13 (ObjNum 177 snap 33 seq_num 177) 2024-03-27T09:36:18.218 INFO:tasks.rados.rados.0.smithi170.stdout:321: done (2 left) 2024-03-27T09:36:18.218 INFO:tasks.rados.rados.0.smithi170.stdout:322: done (1 left) 2024-03-27T09:36:18.218 INFO:tasks.rados.rados.0.smithi170.stdout:324: done (0 left) 2024-03-27T09:36:18.218 INFO:tasks.rados.rados.0.smithi170.stdout:325: delete oid 46 current snap is 33 2024-03-27T09:36:18.225 INFO:tasks.rados.rados.0.smithi170.stdout:325: done (0 left) 2024-03-27T09:36:18.225 INFO:tasks.rados.rados.0.smithi170.stdout:326: write oid 26 current snap is 33 2024-03-27T09:36:18.225 INFO:tasks.rados.rados.0.smithi170.stdout:326: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-03-27T09:36:18.233 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17031214-26 from 642067 to 1338255 tid 1 2024-03-27T09:36:18.239 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17031214-26 from 2025476 to 2484873 tid 2 2024-03-27T09:36:18.245 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17031214-26 from 2971091 to 3042067 tid 3 2024-03-27T09:36:18.245 INFO:tasks.rados.rados.0.smithi170.stdout:327: snap_remove snap 29 2024-03-27T09:36:19.078 INFO:tasks.rados.rados.0.smithi170.stdout:327: done (1 left) 2024-03-27T09:36:19.078 INFO:tasks.rados.rados.0.smithi170.stdout:328: snap_create 2024-03-27T09:36:19.078 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 1 to smithi17031214-26 2024-03-27T09:36:19.078 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 2 to smithi17031214-26 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 3 to smithi17031214-26 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 4 to smithi17031214-26 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 5 to smithi17031214-26 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 26 updating version 0 to 52 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 26 updating version 52 to 53 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 26 updating version 53 to 54 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 26 updating version 54 to 55 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 55 (ObjNum 181 snap 33 seq_num 181) dirty exists 2024-03-27T09:36:19.079 INFO:tasks.rados.rados.0.smithi170.stdout:326: left oid 26 (ObjNum 181 snap 33 seq_num 181) 2024-03-27T09:36:19.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:19.721 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:19.719+0000 7f63a9f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:19.725 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:19.725+0000 7f63a9f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:19.749 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:19.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:20.081 INFO:tasks.rados.rados.0.smithi170.stdout:326: done (1 left) 2024-03-27T09:36:20.082 INFO:tasks.rados.rados.0.smithi170.stdout:328: done (0 left) 2024-03-27T09:36:20.082 INFO:tasks.rados.rados.0.smithi170.stdout:329: write (excl) oid 29 current snap is 34 2024-03-27T09:36:20.082 INFO:tasks.rados.rados.0.smithi170.stdout:329: seq_num 182 ranges {658874=771437,2083958=603055} 2024-03-27T09:36:20.089 INFO:tasks.rados.rados.0.smithi170.stdout:329: writing smithi17031214-29 from 658874 to 1430311 tid 1 2024-03-27T09:36:20.096 INFO:tasks.rados.rados.0.smithi170.stdout:329: writing smithi17031214-29 from 2083958 to 2687013 tid 2 2024-03-27T09:36:20.096 INFO:tasks.rados.rados.0.smithi170.stdout:330: snap_remove snap 25 2024-03-27T09:36:21.086 INFO:tasks.rados.rados.0.smithi170.stdout:330: done (1 left) 2024-03-27T09:36:21.086 INFO:tasks.rados.rados.0.smithi170.stdout:331: write (excl) oid 28 current snap is 34 2024-03-27T09:36:21.086 INFO:tasks.rados.rados.0.smithi170.stdout:329: finishing write tid 1 to smithi17031214-29 2024-03-27T09:36:21.086 INFO:tasks.rados.rados.0.smithi170.stdout:331: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-03-27T09:36:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:331: writing smithi17031214-28 from 675681 to 1238720 tid 1 2024-03-27T09:36:21.100 INFO:tasks.rados.rados.0.smithi170.stdout:331: writing smithi17031214-28 from 1858793 to 2605506 tid 2 2024-03-27T09:36:21.104 INFO:tasks.rados.rados.0.smithi170.stdout:331: writing smithi17031214-28 from 3025937 to 3075681 tid 3 2024-03-27T09:36:21.104 INFO:tasks.rados.rados.0.smithi170.stdout:332: write oid 19 current snap is 34 2024-03-27T09:36:21.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: finishing write tid 2 to smithi17031214-29 2024-03-27T09:36:21.104 INFO:tasks.rados.rados.0.smithi170.stdout:332: seq_num 184 ranges {692488=638288,1917275=606724} 2024-03-27T09:36:21.112 INFO:tasks.rados.rados.0.smithi170.stdout:332: writing smithi17031214-19 from 692488 to 1330776 tid 1 2024-03-27T09:36:21.119 INFO:tasks.rados.rados.0.smithi170.stdout:332: writing smithi17031214-19 from 1917275 to 2523999 tid 2 2024-03-27T09:36:21.119 INFO:tasks.rados.rados.0.smithi170.stdout:333: read oid 41 snap 32 2024-03-27T09:36:21.119 INFO:tasks.rados.rados.0.smithi170.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-27T09:36:21.119 INFO:tasks.rados.rados.0.smithi170.stdout:334: snap_remove snap 33 2024-03-27T09:36:21.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:21.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:21.908 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:21.906+0000 7f445c806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:21.912 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:21.911+0000 7f445c806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:21.937 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:21.943 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:21.946 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:21.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:22.088 INFO:tasks.rados.rados.0.smithi170.stdout:334: done (4 left) 2024-03-27T09:36:22.089 INFO:tasks.rados.rados.0.smithi170.stdout:335: write oid 18 current snap is 34 2024-03-27T09:36:22.089 INFO:tasks.rados.rados.0.smithi170.stdout:329: finishing write tid 3 to smithi17031214-29 2024-03-27T09:36:22.089 INFO:tasks.rados.rados.0.smithi170.stdout:335: seq_num 185 ranges {709295=713537,1859404=750382} 2024-03-27T09:36:22.096 INFO:tasks.rados.rados.0.smithi170.stdout:335: writing smithi17031214-18 from 709295 to 1422832 tid 1 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:335: writing smithi17031214-18 from 1859404 to 2609786 tid 2 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:336: read oid 47 snap -1 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:336: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:337: write (excl) oid 17 current snap is 34 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: finishing write tid 4 to smithi17031214-29 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: oid 29 updating version 0 to 61 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: oid 29 version 61 is already newer than 59 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: oid 29 version 61 is already newer than 61 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 61 (ObjNum 182 snap 34 seq_num 182) dirty exists 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:329: left oid 29 (ObjNum 182 snap 34 seq_num 182) 2024-03-27T09:36:22.104 INFO:tasks.rados.rados.0.smithi170.stdout:337: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-03-27T09:36:22.113 INFO:tasks.rados.rados.0.smithi170.stdout:337: writing smithi17031214-17 from 726102 to 1514888 tid 1 2024-03-27T09:36:22.120 INFO:tasks.rados.rados.0.smithi170.stdout:337: writing smithi17031214-17 from 1917886 to 2528279 tid 2 2024-03-27T09:36:22.123 INFO:tasks.rados.rados.0.smithi170.stdout:337: writing smithi17031214-17 from 3108206 to 3126102 tid 3 2024-03-27T09:36:22.124 INFO:tasks.rados.rados.0.smithi170.stdout:329: done (6 left) 2024-03-27T09:36:22.124 INFO:tasks.rados.rados.0.smithi170.stdout:338: snap_remove snap 28 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:338: done (6 left) 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:339: read oid 1 snap -1 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:339: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:340: snap_create 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:331: finishing write tid 1 to smithi17031214-28 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:332: finishing write tid 1 to smithi17031214-19 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:332: finishing write tid 2 to smithi17031214-19 2024-03-27T09:36:23.093 INFO:tasks.rados.rados.0.smithi170.stdout:331: finishing write tid 2 to smithi17031214-28 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: finishing write tid 3 to smithi17031214-19 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: finishing write tid 3 to smithi17031214-28 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: finishing write tid 4 to smithi17031214-19 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: oid 19 updating version 0 to 58 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: oid 19 version 58 is already newer than 58 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: oid 19 updating version 58 to 59 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 59 (ObjNum 184 snap 34 seq_num 184) dirty exists 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:332: left oid 19 (ObjNum 184 snap 34 seq_num 184) 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: finishing write tid 4 to smithi17031214-28 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: finishing write tid 5 to smithi17031214-28 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: oid 28 updating version 0 to 91 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: oid 28 version 91 is already newer than 90 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: oid 28 version 91 is already newer than 90 2024-03-27T09:36:23.094 INFO:tasks.rados.rados.0.smithi170.stdout:331: oid 28 version 91 is already newer than 88 2024-03-27T09:36:23.095 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 91 (ObjNum 183 snap 34 seq_num 183) dirty exists 2024-03-27T09:36:23.095 INFO:tasks.rados.rados.0.smithi170.stdout:331: left oid 28 (ObjNum 183 snap 34 seq_num 183) 2024-03-27T09:36:23.095 INFO:tasks.rados.rados.0.smithi170.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-27T09:36:23.291 INFO:tasks.rados.rados.0.smithi170.stdout:336: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: finishing write tid 1 to smithi17031214-18 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:337: finishing write tid 1 to smithi17031214-17 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: finishing write tid 2 to smithi17031214-18 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: finishing write tid 3 to smithi17031214-18 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:337: finishing write tid 2 to smithi17031214-17 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: finishing write tid 4 to smithi17031214-18 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: oid 18 updating version 0 to 96 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: oid 18 version 96 is already newer than 96 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:335: oid 18 updating version 96 to 97 2024-03-27T09:36:23.395 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 97 (ObjNum 185 snap 34 seq_num 185) dirty exists 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:335: left oid 18 (ObjNum 185 snap 34 seq_num 185) 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: finishing write tid 3 to smithi17031214-17 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: finishing write tid 4 to smithi17031214-17 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: finishing write tid 5 to smithi17031214-17 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: oid 17 updating version 0 to 103 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: oid 17 version 103 is already newer than 100 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: oid 17 version 103 is already newer than 102 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: oid 17 version 103 is already newer than 101 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 103 (ObjNum 186 snap 34 seq_num 186) dirty exists 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:337: left oid 17 (ObjNum 186 snap 34 seq_num 186) 2024-03-27T09:36:23.396 INFO:tasks.rados.rados.0.smithi170.stdout:339: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:36:23.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2024-03-27T09:36:23.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:24.095 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:24.092+0000 7f953182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:24.095 INFO:tasks.rados.rados.0.smithi170.stdout:331: done (7 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:332: done (6 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:333: done (5 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:335: done (4 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:336: done (3 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:337: done (2 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:339: done (1 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:340: done (0 left) 2024-03-27T09:36:24.096 INFO:tasks.rados.rados.0.smithi170.stdout:341: snap_create 2024-03-27T09:36:24.099 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:24.099+0000 7f953182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:24.155 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:24.218 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:25.097 INFO:tasks.rados.rados.0.smithi170.stdout:341: done (0 left) 2024-03-27T09:36:25.097 INFO:tasks.rados.rados.0.smithi170.stdout:342: read oid 24 snap 32 2024-03-27T09:36:25.097 INFO:tasks.rados.rados.0.smithi170.stdout:342: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-27T09:36:25.097 INFO:tasks.rados.rados.0.smithi170.stdout:343: delete oid 27 current snap is 36 2024-03-27T09:36:25.146 INFO:tasks.rados.rados.0.smithi170.stdout:342: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-27T09:36:25.238 INFO:tasks.rados.rados.0.smithi170.stdout:342: done (1 left) 2024-03-27T09:36:25.238 INFO:tasks.rados.rados.0.smithi170.stdout:343: done (0 left) 2024-03-27T09:36:25.238 INFO:tasks.rados.rados.0.smithi170.stdout:344: write (excl) oid 14 current snap is 36 2024-03-27T09:36:25.238 INFO:tasks.rados.rados.0.smithi170.stdout:344: seq_num 188 ranges {759716=539284,2034850=614062} 2024-03-27T09:36:25.244 INFO:tasks.rados.rados.0.smithi170.stdout:344: writing smithi17031214-14 from 759716 to 1299000 tid 1 2024-03-27T09:36:25.252 INFO:tasks.rados.rados.0.smithi170.stdout:344: writing smithi17031214-14 from 2034850 to 2648912 tid 2 2024-03-27T09:36:25.253 INFO:tasks.rados.rados.0.smithi170.stdout:345: snap_create 2024-03-27T09:36:25.280 INFO:tasks.rados.rados.0.smithi170.stdout:344: finishing write tid 1 to smithi17031214-14 2024-03-27T09:36:25.296 INFO:tasks.rados.rados.0.smithi170.stdout:344: finishing write tid 2 to smithi17031214-14 2024-03-27T09:36:25.306 INFO:tasks.rados.rados.0.smithi170.stdout:344: finishing write tid 3 to smithi17031214-14 2024-03-27T09:36:25.308 INFO:tasks.rados.rados.0.smithi170.stdout:344: finishing write tid 4 to smithi17031214-14 2024-03-27T09:36:25.308 INFO:tasks.rados.rados.0.smithi170.stdout:344: oid 14 updating version 0 to 41 2024-03-27T09:36:25.308 INFO:tasks.rados.rados.0.smithi170.stdout:344: oid 14 updating version 41 to 43 2024-03-27T09:36:25.308 INFO:tasks.rados.rados.0.smithi170.stdout:344: oid 14 version 43 is already newer than 42 2024-03-27T09:36:25.308 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 43 (ObjNum 188 snap 36 seq_num 188) dirty exists 2024-03-27T09:36:25.309 INFO:tasks.rados.rados.0.smithi170.stdout:344: left oid 14 (ObjNum 188 snap 36 seq_num 188) 2024-03-27T09:36:25.944 INFO:tasks.rados.rados.0.smithi170.stdout:344: done (1 left) 2024-03-27T09:36:25.945 INFO:tasks.rados.rados.0.smithi170.stdout:345: done (0 left) 2024-03-27T09:36:25.945 INFO:tasks.rados.rados.0.smithi170.stdout:346: write (excl) oid 13 current snap is 37 2024-03-27T09:36:25.945 INFO:tasks.rados.rados.0.smithi170.stdout:346: seq_num 189 ranges {776523=614533,1976979=757720} 2024-03-27T09:36:25.951 INFO:tasks.rados.rados.0.smithi170.stdout:346: writing smithi17031214-13 from 776523 to 1391056 tid 1 2024-03-27T09:36:25.959 INFO:tasks.rados.rados.0.smithi170.stdout:346: writing smithi17031214-13 from 1976979 to 2734699 tid 2 2024-03-27T09:36:25.960 INFO:tasks.rados.rados.0.smithi170.stdout:347: snap_create 2024-03-27T09:36:26.031 INFO:tasks.rados.rados.0.smithi170.stdout:346: finishing write tid 1 to smithi17031214-13 2024-03-27T09:36:26.047 INFO:tasks.rados.rados.0.smithi170.stdout:346: finishing write tid 2 to smithi17031214-13 2024-03-27T09:36:26.052 INFO:tasks.rados.rados.0.smithi170.stdout:346: finishing write tid 3 to smithi17031214-13 2024-03-27T09:36:26.053 INFO:tasks.rados.rados.0.smithi170.stdout:346: finishing write tid 4 to smithi17031214-13 2024-03-27T09:36:26.053 INFO:tasks.rados.rados.0.smithi170.stdout:346: oid 13 updating version 0 to 47 2024-03-27T09:36:26.054 INFO:tasks.rados.rados.0.smithi170.stdout:346: oid 13 version 47 is already newer than 47 2024-03-27T09:36:26.054 INFO:tasks.rados.rados.0.smithi170.stdout:346: oid 13 version 47 is already newer than 45 2024-03-27T09:36:26.054 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 47 (ObjNum 189 snap 37 seq_num 189) dirty exists 2024-03-27T09:36:26.054 INFO:tasks.rados.rados.0.smithi170.stdout:346: left oid 13 (ObjNum 189 snap 37 seq_num 189) 2024-03-27T09:36:26.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:26.330 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:26.328+0000 7f8d8f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:26.334 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:26.334+0000 7f8d8f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:26.358 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:26.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:346: done (1 left) 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:347: done (0 left) 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:348: read oid 49 snap -1 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:349: read oid 33 snap 37 2024-03-27T09:36:26.946 INFO:tasks.rados.rados.0.smithi170.stdout:349: expect deleted 2024-03-27T09:36:26.947 INFO:tasks.rados.rados.0.smithi170.stdout:350: read oid 38 snap -1 2024-03-27T09:36:26.947 INFO:tasks.rados.rados.0.smithi170.stdout:350: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:36:26.947 INFO:tasks.rados.rados.0.smithi170.stdout:351: snap_remove snap 35 2024-03-27T09:36:26.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:27.062 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:27.065 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:27.947 INFO:tasks.rados.rados.0.smithi170.stdout:351: done (3 left) 2024-03-27T09:36:27.947 INFO:tasks.rados.rados.0.smithi170.stdout:352: snap_remove snap 31 2024-03-27T09:36:28.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:28.523 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:28.521+0000 7f9e2735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:28.527 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:28.526+0000 7f9e2735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:28.551 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:28.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:28.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~39s 2024-03-27T09:36:28.947 INFO:tasks.rados.rados.0.smithi170.stdout:352: done (3 left) 2024-03-27T09:36:28.947 INFO:tasks.rados.rados.0.smithi170.stdout:353: snap_remove snap 36 2024-03-27T09:36:29.949 INFO:tasks.rados.rados.0.smithi170.stdout:353: done (3 left) 2024-03-27T09:36:29.949 INFO:tasks.rados.rados.0.smithi170.stdout:354: delete oid 15 current snap is 38 2024-03-27T09:36:29.949 INFO:tasks.rados.rados.0.smithi170.stdout:350: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:36:30.032 INFO:tasks.rados.rados.0.smithi170.stdout:348: expect (ObjNum 143 snap 18 seq_num 143) 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:348: done (3 left) 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:349: done (2 left) 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:350: done (1 left) 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:354: done (0 left) 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:355: write (excl) oid 24 current snap is 38 2024-03-27T09:36:30.161 INFO:tasks.rados.rados.0.smithi170.stdout:355: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-03-27T09:36:30.165 INFO:tasks.rados.rados.0.smithi170.stdout:355: writing smithi17031214-24 from 410137 to 891521 tid 1 2024-03-27T09:36:30.172 INFO:tasks.rados.rados.0.smithi170.stdout:355: writing smithi17031214-24 from 1410296 to 2171685 tid 2 2024-03-27T09:36:30.178 INFO:tasks.rados.rados.0.smithi170.stdout:355: writing smithi17031214-24 from 2728968 to 3210137 tid 3 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:356: read oid 7 snap -1 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:357: read oid 14 snap -1 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:357: expect (ObjNum 188 snap 36 seq_num 188) 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:358: write oid 36 current snap is 38 2024-03-27T09:36:30.179 INFO:tasks.rados.rados.0.smithi170.stdout:358: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-03-27T09:36:30.184 INFO:tasks.rados.rados.0.smithi170.stdout:358: writing smithi17031214-36 from 426944 to 983577 tid 1 2024-03-27T09:36:30.190 INFO:tasks.rados.rados.0.smithi170.stdout:358: writing smithi17031214-36 from 1468778 to 2090178 tid 2 2024-03-27T09:36:30.198 INFO:tasks.rados.rados.0.smithi170.stdout:358: writing smithi17031214-36 from 2872744 to 3226944 tid 3 2024-03-27T09:36:30.198 INFO:tasks.rados.rados.0.smithi170.stdout:359: read oid 4 snap -1 2024-03-27T09:36:30.198 INFO:tasks.rados.rados.0.smithi170.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2024-03-27T09:36:30.198 INFO:tasks.rados.rados.0.smithi170.stdout:360: read oid 45 snap -1 2024-03-27T09:36:30.199 INFO:tasks.rados.rados.0.smithi170.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2024-03-27T09:36:30.199 INFO:tasks.rados.rados.0.smithi170.stdout:361: write oid 47 current snap is 38 2024-03-27T09:36:30.199 INFO:tasks.rados.rados.0.smithi170.stdout:361: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-03-27T09:36:30.204 INFO:tasks.rados.rados.0.smithi170.stdout:361: writing smithi17031214-47 from 443751 to 1075633 tid 1 2024-03-27T09:36:30.216 INFO:tasks.rados.rados.0.smithi170.stdout:361: writing smithi17031214-47 from 1810907 to 2575965 tid 2 2024-03-27T09:36:30.221 INFO:tasks.rados.rados.0.smithi170.stdout:361: writing smithi17031214-47 from 3067461 to 3243751 tid 3 2024-03-27T09:36:30.221 INFO:tasks.rados.rados.0.smithi170.stdout:362: snap_create 2024-03-27T09:36:30.221 INFO:tasks.rados.rados.0.smithi170.stdout:355: finishing write tid 1 to smithi17031214-24 2024-03-27T09:36:30.261 INFO:tasks.rados.rados.0.smithi170.stdout:355: finishing write tid 2 to smithi17031214-24 2024-03-27T09:36:30.267 INFO:tasks.rados.rados.0.smithi170.stdout:357: expect (ObjNum 188 snap 36 seq_num 188) 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: finishing write tid 3 to smithi17031214-24 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:361: finishing write tid 1 to smithi17031214-47 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: finishing write tid 4 to smithi17031214-24 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: finishing write tid 5 to smithi17031214-24 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: oid 24 updating version 0 to 25 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: oid 24 updating version 25 to 27 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: oid 24 version 27 is already newer than 27 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:355: oid 24 updating version 27 to 28 2024-03-27T09:36:30.458 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 28 (ObjNum 191 snap 38 seq_num 191) dirty exists 2024-03-27T09:36:30.459 INFO:tasks.rados.rados.0.smithi170.stdout:355: left oid 24 (ObjNum 191 snap 38 seq_num 191) 2024-03-27T09:36:30.459 INFO:tasks.rados.rados.0.smithi170.stdout:361: finishing write tid 2 to smithi17031214-47 2024-03-27T09:36:30.459 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 1 to smithi17031214-36 2024-03-27T09:36:30.459 INFO:tasks.rados.rados.0.smithi170.stdout:356: expect (ObjNum 167 snap 26 seq_num 167) 2024-03-27T09:36:30.600 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:30.613 INFO:tasks.rados.rados.0.smithi170.stdout:361: finishing write tid 3 to smithi17031214-47 2024-03-27T09:36:30.613 INFO:tasks.rados.rados.0.smithi170.stdout:360: expect (ObjNum 152 snap 21 seq_num 152) 2024-03-27T09:36:30.709 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:30.707+0000 7fb4aae64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:30.713 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:30.712+0000 7fb4aae64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:30.737 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: finishing write tid 4 to smithi17031214-47 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 2 to smithi17031214-36 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: finishing write tid 5 to smithi17031214-47 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: oid 47 updating version 0 to 40 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: oid 47 version 40 is already newer than 38 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: oid 47 version 40 is already newer than 39 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:361: oid 47 updating version 40 to 41 2024-03-27T09:36:30.748 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 41 (ObjNum 193 snap 38 seq_num 193) dirty exists 2024-03-27T09:36:30.749 INFO:tasks.rados.rados.0.smithi170.stdout:361: left oid 47 (ObjNum 193 snap 38 seq_num 193) 2024-03-27T09:36:30.749 INFO:tasks.rados.rados.0.smithi170.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2024-03-27T09:36:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:30.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 3 to smithi17031214-36 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 4 to smithi17031214-36 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 5 to smithi17031214-36 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 36 updating version 0 to 109 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 36 updating version 109 to 111 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 36 updating version 111 to 112 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 36 version 112 is already newer than 110 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 112 (ObjNum 192 snap 38 seq_num 192) dirty exists 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:358: left oid 36 (ObjNum 192 snap 38 seq_num 192) 2024-03-27T09:36:30.956 INFO:tasks.rados.rados.0.smithi170.stdout:355: done (7 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:356: done (6 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:357: done (5 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:358: done (4 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:359: done (3 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:360: done (2 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:361: done (1 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:362: done (0 left) 2024-03-27T09:36:30.957 INFO:tasks.rados.rados.0.smithi170.stdout:363: snap_remove snap 34 2024-03-27T09:36:31.961 INFO:tasks.rados.rados.0.smithi170.stdout:363: done (0 left) 2024-03-27T09:36:31.961 INFO:tasks.rados.rados.0.smithi170.stdout:364: snap_create 2024-03-27T09:36:32.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:32.167 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:32.170 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:32.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:32.895 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:32.893+0000 7f6203c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:32.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:32.898+0000 7f6203c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:32.922 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:32.962 INFO:tasks.rados.rados.0.smithi170.stdout:364: done (0 left) 2024-03-27T09:36:32.962 INFO:tasks.rados.rados.0.smithi170.stdout:365: write (excl) oid 4 current snap is 40 2024-03-27T09:36:32.963 INFO:tasks.rados.rados.0.smithi170.stdout:365: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-03-27T09:36:32.968 INFO:tasks.rados.rados.0.smithi170.stdout:365: writing smithi17031214-4 from 460558 to 1167689 tid 1 2024-03-27T09:36:32.974 INFO:tasks.rados.rados.0.smithi170.stdout:365: writing smithi17031214-4 from 1869389 to 2378105 tid 2 2024-03-27T09:36:32.979 INFO:tasks.rados.rados.0.smithi170.stdout:365: writing smithi17031214-4 from 3094884 to 3260558 tid 3 2024-03-27T09:36:32.979 INFO:tasks.rados.rados.0.smithi170.stdout:366: write (excl) oid 22 current snap is 40 2024-03-27T09:36:32.979 INFO:tasks.rados.rados.0.smithi170.stdout:366: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-03-27T09:36:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:32.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:32.986 INFO:tasks.rados.rados.0.smithi170.stdout:366: writing smithi17031214-22 from 477365 to 1259745 tid 1 2024-03-27T09:36:32.994 INFO:tasks.rados.rados.0.smithi170.stdout:366: writing smithi17031214-22 from 1927871 to 2696598 tid 2 2024-03-27T09:36:32.998 INFO:tasks.rados.rados.0.smithi170.stdout:366: writing smithi17031214-22 from 3122307 to 3277365 tid 3 2024-03-27T09:36:32.998 INFO:tasks.rados.rados.0.smithi170.stdout:367: write oid 34 current snap is 40 2024-03-27T09:36:32.998 INFO:tasks.rados.rados.0.smithi170.stdout:367: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-03-27T09:36:33.004 INFO:tasks.rados.rados.0.smithi170.stdout:367: writing smithi17031214-34 from 494172 to 951801 tid 1 2024-03-27T09:36:33.012 INFO:tasks.rados.rados.0.smithi170.stdout:367: writing smithi17031214-34 from 1586353 to 2098738 tid 2 2024-03-27T09:36:33.020 INFO:tasks.rados.rados.0.smithi170.stdout:367: writing smithi17031214-34 from 2749730 to 3294172 tid 3 2024-03-27T09:36:33.020 INFO:tasks.rados.rados.0.smithi170.stdout:368: write oid 15 current snap is 40 2024-03-27T09:36:33.020 INFO:tasks.rados.rados.0.smithi170.stdout:368: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-03-27T09:36:33.026 INFO:tasks.rados.rados.0.smithi170.stdout:368: writing smithi17031214-15 from 510979 to 1043857 tid 1 2024-03-27T09:36:33.036 INFO:tasks.rados.rados.0.smithi170.stdout:368: writing smithi17031214-15 from 1528482 to 2300878 tid 2 2024-03-27T09:36:33.043 INFO:tasks.rados.rados.0.smithi170.stdout:368: writing smithi17031214-15 from 3060800 to 3310979 tid 3 2024-03-27T09:36:33.043 INFO:tasks.rados.rados.0.smithi170.stdout:369: read oid 10 snap 38 2024-03-27T09:36:33.044 INFO:tasks.rados.rados.0.smithi170.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2024-03-27T09:36:33.044 INFO:tasks.rados.rados.0.smithi170.stdout:370: write (excl) oid 12 current snap is 40 2024-03-27T09:36:33.044 INFO:tasks.rados.rados.0.smithi170.stdout:370: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-03-27T09:36:33.053 INFO:tasks.rados.rados.0.smithi170.stdout:370: writing smithi17031214-12 from 527786 to 1252266 tid 1 2024-03-27T09:36:33.061 INFO:tasks.rados.rados.0.smithi170.stdout:370: writing smithi17031214-12 from 1703317 to 2219371 tid 2 2024-03-27T09:36:33.068 INFO:tasks.rados.rados.0.smithi170.stdout:370: writing smithi17031214-12 from 2804576 to 3239611 tid 3 2024-03-27T09:36:33.069 INFO:tasks.rados.rados.0.smithi170.stdout:371: read oid 37 snap 38 2024-03-27T09:36:33.069 INFO:tasks.rados.rados.0.smithi170.stdout:371: expect deleted 2024-03-27T09:36:33.069 INFO:tasks.rados.rados.0.smithi170.stdout:372: write (excl) oid 19 current snap is 40 2024-03-27T09:36:33.069 INFO:tasks.rados.rados.0.smithi170.stdout:372: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-03-27T09:36:33.078 INFO:tasks.rados.rados.0.smithi170.stdout:372: writing smithi17031214-19 from 544593 to 1344322 tid 1 2024-03-27T09:36:33.085 INFO:tasks.rados.rados.0.smithi170.stdout:372: writing smithi17031214-19 from 1761799 to 2537864 tid 2 2024-03-27T09:36:33.091 INFO:tasks.rados.rados.0.smithi170.stdout:372: writing smithi17031214-19 from 2948352 to 3344593 tid 3 2024-03-27T09:36:33.091 INFO:tasks.rados.rados.0.smithi170.stdout:373: snap_create 2024-03-27T09:36:33.138 INFO:tasks.rados.rados.0.smithi170.stdout:369: expect (ObjNum 147 snap 20 seq_num 147) 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:366: finishing write tid 1 to smithi17031214-22 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:365: finishing write tid 1 to smithi17031214-4 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:370: finishing write tid 1 to smithi17031214-12 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:368: finishing write tid 1 to smithi17031214-15 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing write tid 1 to smithi17031214-34 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing write tid 1 to smithi17031214-19 2024-03-27T09:36:33.242 INFO:tasks.rados.rados.0.smithi170.stdout:370: finishing write tid 2 to smithi17031214-12 2024-03-27T09:36:33.243 INFO:tasks.rados.rados.0.smithi170.stdout:365: finishing write tid 2 to smithi17031214-4 2024-03-27T09:36:33.243 INFO:tasks.rados.rados.0.smithi170.stdout:366: finishing write tid 2 to smithi17031214-22 2024-03-27T09:36:33.258 INFO:tasks.rados.rados.0.smithi170.stdout:365: finishing write tid 3 to smithi17031214-4 2024-03-27T09:36:33.268 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing write tid 2 to smithi17031214-19 2024-03-27T09:36:33.280 INFO:tasks.rados.rados.0.smithi170.stdout:368: finishing write tid 2 to smithi17031214-15 2024-03-27T09:36:33.282 INFO:tasks.rados.rados.0.smithi170.stdout:370: finishing write tid 3 to smithi17031214-12 2024-03-27T09:36:33.286 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing write tid 2 to smithi17031214-34 2024-03-27T09:36:33.300 INFO:tasks.rados.rados.0.smithi170.stdout:365: finishing write tid 4 to smithi17031214-4 2024-03-27T09:36:33.300 INFO:tasks.rados.rados.0.smithi170.stdout:366: finishing write tid 3 to smithi17031214-22 2024-03-27T09:36:33.301 INFO:tasks.rados.rados.0.smithi170.stdout:365: finishing write tid 5 to smithi17031214-4 2024-03-27T09:36:33.301 INFO:tasks.rados.rados.0.smithi170.stdout:365: oid 4 updating version 0 to 11 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:365: oid 4 updating version 11 to 12 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:365: oid 4 version 12 is already newer than 11 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:365: oid 4 version 12 is already newer than 9 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 12 (ObjNum 194 snap 40 seq_num 194) dirty exists 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:365: left oid 4 (ObjNum 194 snap 40 seq_num 194) 2024-03-27T09:36:33.302 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing write tid 3 to smithi17031214-19 2024-03-27T09:36:33.311 INFO:tasks.rados.rados.0.smithi170.stdout:366: finishing write tid 4 to smithi17031214-22 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: finishing write tid 5 to smithi17031214-22 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: oid 22 updating version 0 to 49 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: oid 22 version 49 is already newer than 47 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: oid 22 version 49 is already newer than 49 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: oid 22 version 49 is already newer than 46 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 49 (ObjNum 195 snap 40 seq_num 195) dirty exists 2024-03-27T09:36:33.314 INFO:tasks.rados.rados.0.smithi170.stdout:366: left oid 22 (ObjNum 195 snap 40 seq_num 195) 2024-03-27T09:36:33.322 INFO:tasks.rados.rados.0.smithi170.stdout:370: finishing write tid 4 to smithi17031214-12 2024-03-27T09:36:33.324 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing write tid 3 to smithi17031214-34 2024-03-27T09:36:33.325 INFO:tasks.rados.rados.0.smithi170.stdout:368: finishing write tid 3 to smithi17031214-15 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing write tid 4 to smithi17031214-19 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: finishing write tid 5 to smithi17031214-12 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: oid 12 updating version 0 to 72 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: oid 12 updating version 72 to 75 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: oid 12 version 75 is already newer than 73 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: oid 12 version 75 is already newer than 74 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 75 (ObjNum 198 snap 40 seq_num 198) dirty exists 2024-03-27T09:36:33.328 INFO:tasks.rados.rados.0.smithi170.stdout:370: left oid 12 (ObjNum 198 snap 40 seq_num 198) 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing write tid 5 to smithi17031214-19 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: oid 19 updating version 0 to 70 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: oid 19 version 70 is already newer than 69 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: oid 19 version 70 is already newer than 67 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: oid 19 version 70 is already newer than 68 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 70 (ObjNum 199 snap 40 seq_num 199) dirty exists 2024-03-27T09:36:33.329 INFO:tasks.rados.rados.0.smithi170.stdout:372: left oid 19 (ObjNum 199 snap 40 seq_num 199) 2024-03-27T09:36:33.330 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing write tid 4 to smithi17031214-34 2024-03-27T09:36:33.331 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing write tid 5 to smithi17031214-34 2024-03-27T09:36:33.331 INFO:tasks.rados.rados.0.smithi170.stdout:367: oid 34 updating version 0 to 116 2024-03-27T09:36:33.331 INFO:tasks.rados.rados.0.smithi170.stdout:367: oid 34 updating version 116 to 119 2024-03-27T09:36:33.331 INFO:tasks.rados.rados.0.smithi170.stdout:367: oid 34 version 119 is already newer than 119 2024-03-27T09:36:33.332 INFO:tasks.rados.rados.0.smithi170.stdout:367: oid 34 version 119 is already newer than 117 2024-03-27T09:36:33.332 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 119 (ObjNum 196 snap 40 seq_num 196) dirty exists 2024-03-27T09:36:33.332 INFO:tasks.rados.rados.0.smithi170.stdout:367: left oid 34 (ObjNum 196 snap 40 seq_num 196) 2024-03-27T09:36:33.335 INFO:tasks.rados.rados.0.smithi170.stdout:368: finishing write tid 4 to smithi17031214-15 2024-03-27T09:36:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:368: finishing write tid 5 to smithi17031214-15 2024-03-27T09:36:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:368: oid 15 updating version 0 to 35 2024-03-27T09:36:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:368: oid 15 version 35 is already newer than 33 2024-03-27T09:36:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:368: oid 15 version 35 is already newer than 34 2024-03-27T09:36:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:368: oid 15 updating version 35 to 36 2024-03-27T09:36:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 36 (ObjNum 197 snap 40 seq_num 197) dirty exists 2024-03-27T09:36:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:368: left oid 15 (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:36:33.967 INFO:tasks.rados.rados.0.smithi170.stdout:365: done (8 left) 2024-03-27T09:36:33.967 INFO:tasks.rados.rados.0.smithi170.stdout:366: done (7 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:367: done (6 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:368: done (5 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:369: done (4 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:370: done (3 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:371: done (2 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:372: done (1 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:373: done (0 left) 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:374: write oid 33 current snap is 41 2024-03-27T09:36:33.968 INFO:tasks.rados.rados.0.smithi170.stdout:374: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-03-27T09:36:33.973 INFO:tasks.rados.rados.0.smithi170.stdout:374: writing smithi17031214-33 from 561400 to 1036378 tid 1 2024-03-27T09:36:33.980 INFO:tasks.rados.rados.0.smithi170.stdout:374: writing smithi17031214-33 from 1820281 to 2340004 tid 2 2024-03-27T09:36:33.987 INFO:tasks.rados.rados.0.smithi170.stdout:374: writing smithi17031214-33 from 2859422 to 3317001 tid 3 2024-03-27T09:36:33.987 INFO:tasks.rados.rados.0.smithi170.stdout:375: read oid 5 snap -1 2024-03-27T09:36:33.987 INFO:tasks.rados.rados.0.smithi170.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2024-03-27T09:36:33.987 INFO:tasks.rados.rados.0.smithi170.stdout:376: read oid 32 snap -1 2024-03-27T09:36:33.987 INFO:tasks.rados.rados.0.smithi170.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2024-03-27T09:36:33.988 INFO:tasks.rados.rados.0.smithi170.stdout:377: read oid 38 snap -1 2024-03-27T09:36:33.988 INFO:tasks.rados.rados.0.smithi170.stdout:377: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:36:33.988 INFO:tasks.rados.rados.0.smithi170.stdout:378: snap_create 2024-03-27T09:36:34.035 INFO:tasks.rados.rados.0.smithi170.stdout:377: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:36:34.126 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 1 to smithi17031214-33 2024-03-27T09:36:34.126 INFO:tasks.rados.rados.0.smithi170.stdout:375: expect (ObjNum 126 snap 15 seq_num 126) 2024-03-27T09:36:34.223 INFO:tasks.rados.rados.0.smithi170.stdout:376: expect (ObjNum 174 snap 29 seq_num 174) 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 2 to smithi17031214-33 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 3 to smithi17031214-33 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 4 to smithi17031214-33 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 5 to smithi17031214-33 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 33 updating version 0 to 69 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 33 updating version 69 to 71 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 33 updating version 71 to 72 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 33 version 72 is already newer than 70 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 72 (ObjNum 200 snap 41 seq_num 200) dirty exists 2024-03-27T09:36:34.333 INFO:tasks.rados.rados.0.smithi170.stdout:374: left oid 33 (ObjNum 200 snap 41 seq_num 200) 2024-03-27T09:36:34.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~44s 2024-03-27T09:36:34.970 INFO:tasks.rados.rados.0.smithi170.stdout:374: done (4 left) 2024-03-27T09:36:34.971 INFO:tasks.rados.rados.0.smithi170.stdout:375: done (3 left) 2024-03-27T09:36:34.971 INFO:tasks.rados.rados.0.smithi170.stdout:376: done (2 left) 2024-03-27T09:36:34.971 INFO:tasks.rados.rados.0.smithi170.stdout:377: done (1 left) 2024-03-27T09:36:34.971 INFO:tasks.rados.rados.0.smithi170.stdout:378: done (0 left) 2024-03-27T09:36:34.971 INFO:tasks.rados.rados.0.smithi170.stdout:379: delete oid 8 current snap is 42 2024-03-27T09:36:34.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:35.023 INFO:tasks.rados.rados.0.smithi170.stdout:379: done (0 left) 2024-03-27T09:36:35.023 INFO:tasks.rados.rados.0.smithi170.stdout:380: read oid 14 snap -1 2024-03-27T09:36:35.023 INFO:tasks.rados.rados.0.smithi170.stdout:380: expect (ObjNum 188 snap 36 seq_num 188) 2024-03-27T09:36:35.023 INFO:tasks.rados.rados.0.smithi170.stdout:381: snap_create 2024-03-27T09:36:35.084 INFO:tasks.rados.rados.0.smithi170.stdout:380: expect (ObjNum 188 snap 36 seq_num 188) 2024-03-27T09:36:35.110 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:35.107+0000 7f2f88354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:35.115 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:35.114+0000 7f2f88354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:35.145 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:35.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:35.946 INFO:tasks.rados.rados.0.smithi170.stdout:380: done (1 left) 2024-03-27T09:36:35.946 INFO:tasks.rados.rados.0.smithi170.stdout:381: done (0 left) 2024-03-27T09:36:35.946 INFO:tasks.rados.rados.0.smithi170.stdout:382: snap_remove snap 40 2024-03-27T09:36:36.948 INFO:tasks.rados.rados.0.smithi170.stdout:382: done (0 left) 2024-03-27T09:36:36.948 INFO:tasks.rados.rados.0.smithi170.stdout:383: write (excl) oid 17 current snap is 43 2024-03-27T09:36:36.948 INFO:tasks.rados.rados.0.smithi170.stdout:383: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-03-27T09:36:36.954 INFO:tasks.rados.rados.0.smithi170.stdout:383: writing smithi17031214-17 from 595014 to 1220490 tid 1 2024-03-27T09:36:36.961 INFO:tasks.rados.rados.0.smithi170.stdout:383: writing smithi17031214-17 from 1820892 to 2344284 tid 2 2024-03-27T09:36:36.967 INFO:tasks.rados.rados.0.smithi170.stdout:383: writing smithi17031214-17 from 2797915 to 3394391 tid 3 2024-03-27T09:36:36.967 INFO:tasks.rados.rados.0.smithi170.stdout:384: read oid 20 snap -1 2024-03-27T09:36:36.967 INFO:tasks.rados.rados.0.smithi170.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2024-03-27T09:36:36.967 INFO:tasks.rados.rados.0.smithi170.stdout:385: delete oid 42 current snap is 43 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:385: done (2 left) 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:386: read oid 12 snap 41 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:386: expect (ObjNum 198 snap 40 seq_num 198) 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:387: read oid 46 snap -1 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:387: expect deleted 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:388: read oid 47 snap -1 2024-03-27T09:36:37.016 INFO:tasks.rados.rados.0.smithi170.stdout:388: expect (ObjNum 193 snap 38 seq_num 193) 2024-03-27T09:36:37.017 INFO:tasks.rados.rados.0.smithi170.stdout:389: read oid 26 snap -1 2024-03-27T09:36:37.017 INFO:tasks.rados.rados.0.smithi170.stdout:389: expect (ObjNum 181 snap 33 seq_num 181) 2024-03-27T09:36:37.017 INFO:tasks.rados.rados.0.smithi170.stdout:390: read oid 43 snap -1 2024-03-27T09:36:37.017 INFO:tasks.rados.rados.0.smithi170.stdout:390: expect deleted 2024-03-27T09:36:37.017 INFO:tasks.rados.rados.0.smithi170.stdout:391: snap_create 2024-03-27T09:36:37.096 INFO:tasks.rados.rados.0.smithi170.stdout:389: expect (ObjNum 181 snap 33 seq_num 181) 2024-03-27T09:36:37.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:37.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:37.216 INFO:tasks.rados.rados.0.smithi170.stdout:384: expect (ObjNum 154 snap 21 seq_num 154) 2024-03-27T09:36:37.271 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:37.275 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:37.323 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:37.321+0000 7f061e0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:37.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:37.326+0000 7f061e0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:37.351 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:37.375 INFO:tasks.rados.rados.0.smithi170.stdout:388: expect (ObjNum 193 snap 38 seq_num 193) 2024-03-27T09:36:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:37.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing write tid 1 to smithi17031214-17 2024-03-27T09:36:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing write tid 2 to smithi17031214-17 2024-03-27T09:36:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:386: expect (ObjNum 198 snap 40 seq_num 198) 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing write tid 3 to smithi17031214-17 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing write tid 4 to smithi17031214-17 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing write tid 5 to smithi17031214-17 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: oid 17 updating version 0 to 117 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: oid 17 updating version 117 to 120 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: oid 17 version 120 is already newer than 118 2024-03-27T09:36:37.662 INFO:tasks.rados.rados.0.smithi170.stdout:383: oid 17 version 120 is already newer than 119 2024-03-27T09:36:37.663 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 120 (ObjNum 202 snap 43 seq_num 202) dirty exists 2024-03-27T09:36:37.663 INFO:tasks.rados.rados.0.smithi170.stdout:383: left oid 17 (ObjNum 202 snap 43 seq_num 202) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:383: done (7 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:384: done (6 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:386: done (5 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:387: done (4 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:388: done (3 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:389: done (2 left) 2024-03-27T09:36:37.977 INFO:tasks.rados.rados.0.smithi170.stdout:390: done (1 left) 2024-03-27T09:36:37.978 INFO:tasks.rados.rados.0.smithi170.stdout:391: done (0 left) 2024-03-27T09:36:37.978 INFO:tasks.rados.rados.0.smithi170.stdout:392: read oid 26 snap 37 2024-03-27T09:36:37.978 INFO:tasks.rados.rados.0.smithi170.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2024-03-27T09:36:37.978 INFO:tasks.rados.rados.0.smithi170.stdout:393: delete oid 29 current snap is 44 2024-03-27T09:36:38.047 INFO:tasks.rados.rados.0.smithi170.stdout:392: expect (ObjNum 181 snap 33 seq_num 181) 2024-03-27T09:36:38.162 INFO:tasks.rados.rados.0.smithi170.stdout:392: done (1 left) 2024-03-27T09:36:38.162 INFO:tasks.rados.rados.0.smithi170.stdout:393: done (0 left) 2024-03-27T09:36:38.162 INFO:tasks.rados.rados.0.smithi170.stdout:394: write oid 4 current snap is 44 2024-03-27T09:36:38.163 INFO:tasks.rados.rados.0.smithi170.stdout:394: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-03-27T09:36:38.168 INFO:tasks.rados.rados.0.smithi170.stdout:394: writing smithi17031214-4 from 645435 to 1096658 tid 1 2024-03-27T09:36:38.177 INFO:tasks.rados.rados.0.smithi170.stdout:394: writing smithi17031214-4 from 1879985 to 2667057 tid 2 2024-03-27T09:36:38.181 INFO:tasks.rados.rados.0.smithi170.stdout:394: writing smithi17031214-4 from 3280184 to 3445435 tid 3 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:395: read oid 29 snap -1 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:395: expect deleted 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:396: read oid 44 snap -1 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:397: write (excl) oid 23 current snap is 44 2024-03-27T09:36:38.182 INFO:tasks.rados.rados.0.smithi170.stdout:397: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-03-27T09:36:38.188 INFO:tasks.rados.rados.0.smithi170.stdout:397: writing smithi17031214-23 from 662242 to 1305067 tid 1 2024-03-27T09:36:38.198 INFO:tasks.rados.rados.0.smithi170.stdout:397: writing smithi17031214-23 from 2054820 to 2585550 tid 2 2024-03-27T09:36:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:397: writing smithi17031214-23 from 3307607 to 3462242 tid 3 2024-03-27T09:36:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:398: delete oid 31 current snap is 44 2024-03-27T09:36:38.206 INFO:tasks.rados.rados.0.smithi170.stdout:394: finishing write tid 1 to smithi17031214-4 2024-03-27T09:36:38.246 INFO:tasks.rados.rados.0.smithi170.stdout:394: finishing write tid 2 to smithi17031214-4 2024-03-27T09:36:38.248 INFO:tasks.rados.rados.0.smithi170.stdout:397: finishing write tid 1 to smithi17031214-23 2024-03-27T09:36:38.250 INFO:tasks.rados.rados.0.smithi170.stdout:396: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:394: finishing write tid 3 to smithi17031214-4 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:395: done (4 left) 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:396: done (3 left) 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:398: done (2 left) 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:399: write oid 48 current snap is 44 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:397: finishing write tid 2 to smithi17031214-23 2024-03-27T09:36:38.480 INFO:tasks.rados.rados.0.smithi170.stdout:394: finishing write tid 4 to smithi17031214-4 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: finishing write tid 5 to smithi17031214-4 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: oid 4 updating version 0 to 17 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: oid 4 version 17 is already newer than 14 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: oid 4 version 17 is already newer than 16 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: oid 4 version 17 is already newer than 15 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 17 (ObjNum 205 snap 44 seq_num 205) dirty exists 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:394: left oid 4 (ObjNum 205 snap 44 seq_num 205) 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:397: finishing write tid 3 to smithi17031214-23 2024-03-27T09:36:38.481 INFO:tasks.rados.rados.0.smithi170.stdout:399: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-03-27T09:36:38.487 INFO:tasks.rados.rados.0.smithi170.stdout:399: writing smithi17031214-48 from 695856 to 1489179 tid 1 2024-03-27T09:36:38.494 INFO:tasks.rados.rados.0.smithi170.stdout:399: writing smithi17031214-48 from 2171784 to 2706183 tid 2 2024-03-27T09:36:38.499 INFO:tasks.rados.rados.0.smithi170.stdout:399: writing smithi17031214-48 from 3362453 to 3495856 tid 3 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:394: done (2 left) 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:400: write oid 32 current snap is 44 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: finishing write tid 4 to smithi17031214-23 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: finishing write tid 5 to smithi17031214-23 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: oid 23 updating version 0 to 124 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: oid 23 version 124 is already newer than 122 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: oid 23 version 124 is already newer than 124 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: oid 23 version 124 is already newer than 121 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 124 (ObjNum 206 snap 44 seq_num 206) dirty exists 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:397: left oid 23 (ObjNum 206 snap 44 seq_num 206) 2024-03-27T09:36:38.500 INFO:tasks.rados.rados.0.smithi170.stdout:400: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-03-27T09:36:38.507 INFO:tasks.rados.rados.0.smithi170.stdout:400: writing smithi17031214-32 from 712663 to 1181235 tid 1 2024-03-27T09:36:38.514 INFO:tasks.rados.rados.0.smithi170.stdout:400: writing smithi17031214-32 from 1713913 to 2391970 tid 2 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:400: writing smithi17031214-32 from 2873523 to 3465256 tid 3 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:397: done (2 left) 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:401: read oid 15 snap -1 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:401: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:402: snap_create 2024-03-27T09:36:38.521 INFO:tasks.rados.rados.0.smithi170.stdout:399: finishing write tid 1 to smithi17031214-48 2024-03-27T09:36:38.540 INFO:tasks.rados.rados.0.smithi170.stdout:400: finishing write tid 1 to smithi17031214-32 2024-03-27T09:36:38.549 INFO:tasks.rados.rados.0.smithi170.stdout:399: finishing write tid 2 to smithi17031214-48 2024-03-27T09:36:38.577 INFO:tasks.rados.rados.0.smithi170.stdout:399: finishing write tid 3 to smithi17031214-48 2024-03-27T09:36:38.578 INFO:tasks.rados.rados.0.smithi170.stdout:400: finishing write tid 2 to smithi17031214-32 2024-03-27T09:36:38.584 INFO:tasks.rados.rados.0.smithi170.stdout:400: finishing write tid 3 to smithi17031214-32 2024-03-27T09:36:38.591 INFO:tasks.rados.rados.0.smithi170.stdout:399: finishing write tid 4 to smithi17031214-48 2024-03-27T09:36:38.593 INFO:tasks.rados.rados.0.smithi170.stdout:399: finishing write tid 5 to smithi17031214-48 2024-03-27T09:36:38.593 INFO:tasks.rados.rados.0.smithi170.stdout:399: oid 48 updating version 0 to 74 2024-03-27T09:36:38.593 INFO:tasks.rados.rados.0.smithi170.stdout:399: oid 48 updating version 74 to 77 2024-03-27T09:36:38.593 INFO:tasks.rados.rados.0.smithi170.stdout:399: oid 48 version 77 is already newer than 76 2024-03-27T09:36:38.593 INFO:tasks.rados.rados.0.smithi170.stdout:399: oid 48 version 77 is already newer than 76 2024-03-27T09:36:38.594 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 77 (ObjNum 208 snap 44 seq_num 208) dirty exists 2024-03-27T09:36:38.594 INFO:tasks.rados.rados.0.smithi170.stdout:399: left oid 48 (ObjNum 208 snap 44 seq_num 208) 2024-03-27T09:36:38.597 INFO:tasks.rados.rados.0.smithi170.stdout:401: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: finishing write tid 4 to smithi17031214-32 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: finishing write tid 5 to smithi17031214-32 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: oid 32 updating version 0 to 107 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: oid 32 updating version 107 to 109 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: oid 32 updating version 109 to 110 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:400: oid 32 version 110 is already newer than 109 2024-03-27T09:36:38.721 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 110 (ObjNum 209 snap 44 seq_num 209) dirty exists 2024-03-27T09:36:38.722 INFO:tasks.rados.rados.0.smithi170.stdout:400: left oid 32 (ObjNum 209 snap 44 seq_num 209) 2024-03-27T09:36:38.980 INFO:tasks.rados.rados.0.smithi170.stdout:399: done (3 left) 2024-03-27T09:36:38.980 INFO:tasks.rados.rados.0.smithi170.stdout:400: done (2 left) 2024-03-27T09:36:38.980 INFO:tasks.rados.rados.0.smithi170.stdout:401: done (1 left) 2024-03-27T09:36:38.980 INFO:tasks.rados.rados.0.smithi170.stdout:402: done (0 left) 2024-03-27T09:36:38.980 INFO:tasks.rados.rados.0.smithi170.stdout:403: read oid 17 snap -1 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:404: read oid 35 snap 37 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:404: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:405: read oid 8 snap -1 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:405: expect deleted 2024-03-27T09:36:38.981 INFO:tasks.rados.rados.0.smithi170.stdout:406: snap_remove snap 44 2024-03-27T09:36:39.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:39.509 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:39.507+0000 7fb09c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:39.512 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:39.512+0000 7fb09c0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:39.536 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:39.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:39.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2024-03-27T09:36:39.983 INFO:tasks.rados.rados.0.smithi170.stdout:406: done (3 left) 2024-03-27T09:36:39.983 INFO:tasks.rados.rados.0.smithi170.stdout:407: delete oid 4 current snap is 45 2024-03-27T09:36:39.984 INFO:tasks.rados.rados.0.smithi170.stdout:404: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:36:40.234 INFO:tasks.rados.rados.0.smithi170.stdout:403: expect (ObjNum 202 snap 43 seq_num 202) 2024-03-27T09:36:40.489 INFO:tasks.rados.rados.0.smithi170.stdout:403: done (3 left) 2024-03-27T09:36:40.489 INFO:tasks.rados.rados.0.smithi170.stdout:404: done (2 left) 2024-03-27T09:36:40.490 INFO:tasks.rados.rados.0.smithi170.stdout:405: done (1 left) 2024-03-27T09:36:40.490 INFO:tasks.rados.rados.0.smithi170.stdout:407: done (0 left) 2024-03-27T09:36:40.490 INFO:tasks.rados.rados.0.smithi170.stdout:408: write oid 13 current snap is 45 2024-03-27T09:36:40.490 INFO:tasks.rados.rados.0.smithi170.stdout:408: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-03-27T09:36:40.496 INFO:tasks.rados.rados.0.smithi170.stdout:408: writing smithi17031214-13 from 746277 to 1365347 tid 1 2024-03-27T09:36:40.505 INFO:tasks.rados.rados.0.smithi170.stdout:408: writing smithi17031214-13 from 1830877 to 2512603 tid 2 2024-03-27T09:36:40.513 INFO:tasks.rados.rados.0.smithi170.stdout:408: writing smithi17031214-13 from 2928369 to 3546277 tid 3 2024-03-27T09:36:40.513 INFO:tasks.rados.rados.0.smithi170.stdout:409: write (excl) oid 14 current snap is 45 2024-03-27T09:36:40.514 INFO:tasks.rados.rados.0.smithi170.stdout:409: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-03-27T09:36:40.523 INFO:tasks.rados.rados.0.smithi170.stdout:409: writing smithi17031214-14 from 763084 to 1457403 tid 1 2024-03-27T09:36:40.530 INFO:tasks.rados.rados.0.smithi170.stdout:409: writing smithi17031214-14 from 1889359 to 2431096 tid 2 2024-03-27T09:36:40.536 INFO:tasks.rados.rados.0.smithi170.stdout:409: writing smithi17031214-14 from 2955792 to 3497694 tid 3 2024-03-27T09:36:40.536 INFO:tasks.rados.rados.0.smithi170.stdout:410: read oid 27 snap -1 2024-03-27T09:36:40.537 INFO:tasks.rados.rados.0.smithi170.stdout:410: expect deleted 2024-03-27T09:36:40.537 INFO:tasks.rados.rados.0.smithi170.stdout:411: snap_remove snap 37 2024-03-27T09:36:40.986 INFO:tasks.rados.rados.0.smithi170.stdout:411: done (3 left) 2024-03-27T09:36:40.986 INFO:tasks.rados.rados.0.smithi170.stdout:412: write (excl) oid 31 current snap is 45 2024-03-27T09:36:40.986 INFO:tasks.rados.rados.0.smithi170.stdout:408: finishing write tid 1 to smithi17031214-13 2024-03-27T09:36:40.986 INFO:tasks.rados.rados.0.smithi170.stdout:412: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-03-27T09:36:40.992 INFO:tasks.rados.rados.0.smithi170.stdout:412: writing smithi17031214-31 from 779891 to 1265812 tid 1 2024-03-27T09:36:41.000 INFO:tasks.rados.rados.0.smithi170.stdout:412: writing smithi17031214-31 from 1947841 to 2633236 tid 2 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:412: writing smithi17031214-31 from 3383215 to 3579891 tid 3 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:413: snap_create 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: finishing write tid 2 to smithi17031214-13 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: finishing write tid 3 to smithi17031214-13 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: finishing write tid 4 to smithi17031214-13 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: finishing write tid 5 to smithi17031214-13 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: oid 13 updating version 0 to 62 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: oid 13 version 62 is already newer than 61 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: oid 13 updating version 62 to 64 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: oid 13 version 64 is already newer than 63 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 64 (ObjNum 211 snap 45 seq_num 211) dirty exists 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:408: left oid 13 (ObjNum 211 snap 45 seq_num 211) 2024-03-27T09:36:41.006 INFO:tasks.rados.rados.0.smithi170.stdout:409: finishing write tid 1 to smithi17031214-14 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: finishing write tid 2 to smithi17031214-14 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: finishing write tid 3 to smithi17031214-14 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: finishing write tid 4 to smithi17031214-14 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: finishing write tid 5 to smithi17031214-14 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: oid 14 updating version 0 to 66 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: oid 14 updating version 66 to 69 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: oid 14 version 69 is already newer than 69 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: oid 14 version 69 is already newer than 69 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 69 (ObjNum 212 snap 45 seq_num 212) dirty exists 2024-03-27T09:36:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:409: left oid 14 (ObjNum 212 snap 45 seq_num 212) 2024-03-27T09:36:41.079 INFO:tasks.rados.rados.0.smithi170.stdout:412: finishing write tid 1 to smithi17031214-31 2024-03-27T09:36:41.135 INFO:tasks.rados.rados.0.smithi170.stdout:412: finishing write tid 2 to smithi17031214-31 2024-03-27T09:36:41.146 INFO:tasks.rados.rados.0.smithi170.stdout:412: finishing write tid 3 to smithi17031214-31 2024-03-27T09:36:41.153 INFO:tasks.rados.rados.0.smithi170.stdout:412: finishing write tid 4 to smithi17031214-31 2024-03-27T09:36:41.155 INFO:tasks.rados.rados.0.smithi170.stdout:412: finishing write tid 5 to smithi17031214-31 2024-03-27T09:36:41.155 INFO:tasks.rados.rados.0.smithi170.stdout:412: oid 31 updating version 0 to 85 2024-03-27T09:36:41.155 INFO:tasks.rados.rados.0.smithi170.stdout:412: oid 31 updating version 85 to 86 2024-03-27T09:36:41.155 INFO:tasks.rados.rados.0.smithi170.stdout:412: oid 31 version 86 is already newer than 84 2024-03-27T09:36:41.155 INFO:tasks.rados.rados.0.smithi170.stdout:412: oid 31 updating version 86 to 87 2024-03-27T09:36:41.156 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 87 (ObjNum 213 snap 45 seq_num 213) dirty exists 2024-03-27T09:36:41.156 INFO:tasks.rados.rados.0.smithi170.stdout:412: left oid 31 (ObjNum 213 snap 45 seq_num 213) 2024-03-27T09:36:41.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:41.693 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:41.690+0000 7f413ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:41.696 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:41.696+0000 7f413ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:41.720 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:41.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:41.990 INFO:tasks.rados.rados.0.smithi170.stdout:408: done (4 left) 2024-03-27T09:36:41.990 INFO:tasks.rados.rados.0.smithi170.stdout:409: done (3 left) 2024-03-27T09:36:41.991 INFO:tasks.rados.rados.0.smithi170.stdout:410: done (2 left) 2024-03-27T09:36:41.991 INFO:tasks.rados.rados.0.smithi170.stdout:412: done (1 left) 2024-03-27T09:36:41.991 INFO:tasks.rados.rados.0.smithi170.stdout:413: done (0 left) 2024-03-27T09:36:41.991 INFO:tasks.rados.rados.0.smithi170.stdout:414: snap_remove snap 43 2024-03-27T09:36:42.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:42.375 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:42.378 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:414: done (0 left) 2024-03-27T09:36:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:415: read oid 22 snap -1 2024-03-27T09:36:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2024-03-27T09:36:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:416: snap_remove snap 38 2024-03-27T09:36:43.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:43.876 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:43.874+0000 7faf732ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:43.880 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:43.879+0000 7faf732ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:43.903 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:43.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:43.995 INFO:tasks.rados.rados.0.smithi170.stdout:416: done (1 left) 2024-03-27T09:36:43.995 INFO:tasks.rados.rados.0.smithi170.stdout:417: write (excl) oid 14 current snap is 46 2024-03-27T09:36:43.995 INFO:tasks.rados.rados.0.smithi170.stdout:417: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-03-27T09:36:44.002 INFO:tasks.rados.rados.0.smithi170.stdout:417: writing smithi17031214-14 from 796698 to 1357868 tid 1 2024-03-27T09:36:44.011 INFO:tasks.rados.rados.0.smithi170.stdout:417: writing smithi17031214-14 from 2006323 to 2551729 tid 2 2024-03-27T09:36:44.020 INFO:tasks.rados.rados.0.smithi170.stdout:417: writing smithi17031214-14 from 3126991 to 3596698 tid 3 2024-03-27T09:36:44.020 INFO:tasks.rados.rados.0.smithi170.stdout:418: write oid 11 current snap is 46 2024-03-27T09:36:44.020 INFO:tasks.rados.rados.0.smithi170.stdout:418: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-03-27T09:36:44.030 INFO:tasks.rados.rados.0.smithi170.stdout:418: writing smithi17031214-11 from 413505 to 1049924 tid 1 2024-03-27T09:36:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:418: writing smithi17031214-11 from 1664805 to 2353869 tid 2 2024-03-27T09:36:44.046 INFO:tasks.rados.rados.0.smithi170.stdout:418: writing smithi17031214-11 from 3038061 to 3530132 tid 3 2024-03-27T09:36:44.046 INFO:tasks.rados.rados.0.smithi170.stdout:419: read oid 30 snap -1 2024-03-27T09:36:44.047 INFO:tasks.rados.rados.0.smithi170.stdout:419: expect deleted 2024-03-27T09:36:44.047 INFO:tasks.rados.rados.0.smithi170.stdout:420: snap_create 2024-03-27T09:36:44.047 INFO:tasks.rados.rados.0.smithi170.stdout:415: expect (ObjNum 195 snap 40 seq_num 195) 2024-03-27T09:36:44.186 INFO:tasks.rados.rados.0.smithi170.stdout:418: finishing write tid 1 to smithi17031214-11 2024-03-27T09:36:44.187 INFO:tasks.rados.rados.0.smithi170.stdout:418: finishing write tid 2 to smithi17031214-11 2024-03-27T09:36:44.187 INFO:tasks.rados.rados.0.smithi170.stdout:418: finishing write tid 3 to smithi17031214-11 2024-03-27T09:36:44.190 INFO:tasks.rados.rados.0.smithi170.stdout:418: finishing write tid 4 to smithi17031214-11 2024-03-27T09:36:44.191 INFO:tasks.rados.rados.0.smithi170.stdout:417: finishing write tid 1 to smithi17031214-14 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: finishing write tid 5 to smithi17031214-11 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: oid 11 updating version 0 to 98 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: oid 11 version 98 is already newer than 98 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: oid 11 updating version 98 to 99 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: oid 11 version 99 is already newer than 96 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 99 (ObjNum 215 snap 46 seq_num 215) dirty exists 2024-03-27T09:36:44.193 INFO:tasks.rados.rados.0.smithi170.stdout:418: left oid 11 (ObjNum 215 snap 46 seq_num 215) 2024-03-27T09:36:44.212 INFO:tasks.rados.rados.0.smithi170.stdout:417: finishing write tid 2 to smithi17031214-14 2024-03-27T09:36:44.224 INFO:tasks.rados.rados.0.smithi170.stdout:417: finishing write tid 3 to smithi17031214-14 2024-03-27T09:36:44.235 INFO:tasks.rados.rados.0.smithi170.stdout:417: finishing write tid 4 to smithi17031214-14 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: finishing write tid 5 to smithi17031214-14 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: oid 14 updating version 0 to 85 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: oid 14 version 85 is already newer than 83 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: oid 14 updating version 85 to 86 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: oid 14 version 86 is already newer than 85 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 86 (ObjNum 214 snap 46 seq_num 214) dirty exists 2024-03-27T09:36:44.239 INFO:tasks.rados.rados.0.smithi170.stdout:417: left oid 14 (ObjNum 214 snap 46 seq_num 214) 2024-03-27T09:36:44.999 INFO:tasks.rados.rados.0.smithi170.stdout:415: done (4 left) 2024-03-27T09:36:44.999 INFO:tasks.rados.rados.0.smithi170.stdout:417: done (3 left) 2024-03-27T09:36:44.999 INFO:tasks.rados.rados.0.smithi170.stdout:418: done (2 left) 2024-03-27T09:36:44.999 INFO:tasks.rados.rados.0.smithi170.stdout:419: done (1 left) 2024-03-27T09:36:45.000 INFO:tasks.rados.rados.0.smithi170.stdout:420: done (0 left) 2024-03-27T09:36:45.000 INFO:tasks.rados.rados.0.smithi170.stdout:421: write (excl) oid 13 current snap is 47 2024-03-27T09:36:45.000 INFO:tasks.rados.rados.0.smithi170.stdout:421: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-03-27T09:36:45.004 INFO:tasks.rados.rados.0.smithi170.stdout:421: writing smithi17031214-13 from 430312 to 1141980 tid 1 2024-03-27T09:36:45.013 INFO:tasks.rados.rados.0.smithi170.stdout:421: writing smithi17031214-13 from 1723287 to 2272362 tid 2 2024-03-27T09:36:45.022 INFO:tasks.rados.rados.0.smithi170.stdout:421: writing smithi17031214-13 from 2781837 to 3485180 tid 3 2024-03-27T09:36:45.023 INFO:tasks.rados.rados.0.smithi170.stdout:422: write oid 48 current snap is 47 2024-03-27T09:36:45.023 INFO:tasks.rados.rados.0.smithi170.stdout:422: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-03-27T09:36:45.033 INFO:tasks.rados.rados.0.smithi170.stdout:422: writing smithi17031214-48 from 447119 to 1234036 tid 1 2024-03-27T09:36:45.040 INFO:tasks.rados.rados.0.smithi170.stdout:422: writing smithi17031214-48 from 1665416 to 2358149 tid 2 2024-03-27T09:36:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:422: writing smithi17031214-48 from 2976554 to 3491169 tid 3 2024-03-27T09:36:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:423: write (excl) oid 12 current snap is 47 2024-03-27T09:36:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:423: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-03-27T09:36:45.052 INFO:tasks.rados.rados.0.smithi170.stdout:423: writing smithi17031214-12 from 463926 to 926092 tid 1 2024-03-27T09:36:45.059 INFO:tasks.rados.rados.0.smithi170.stdout:423: writing smithi17031214-12 from 1723898 to 2160289 tid 2 2024-03-27T09:36:45.066 INFO:tasks.rados.rados.0.smithi170.stdout:423: writing smithi17031214-12 from 2603977 to 3329864 tid 3 2024-03-27T09:36:45.066 INFO:tasks.rados.rados.0.smithi170.stdout:424: snap_remove snap 45 2024-03-27T09:36:45.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-03-27T09:36:45.949 INFO:tasks.rados.rados.0.smithi170.stdout:424: done (3 left) 2024-03-27T09:36:45.949 INFO:tasks.rados.rados.0.smithi170.stdout:425: snap_create 2024-03-27T09:36:45.949 INFO:tasks.rados.rados.0.smithi170.stdout:422: finishing write tid 1 to smithi17031214-48 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:421: finishing write tid 1 to smithi17031214-13 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: finishing write tid 2 to smithi17031214-48 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: finishing write tid 3 to smithi17031214-48 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: finishing write tid 4 to smithi17031214-48 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: finishing write tid 5 to smithi17031214-48 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: oid 48 updating version 0 to 88 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: oid 48 version 88 is already newer than 85 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: oid 48 version 88 is already newer than 88 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: oid 48 version 88 is already newer than 86 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 88 (ObjNum 217 snap 47 seq_num 217) dirty exists 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:422: left oid 48 (ObjNum 217 snap 47 seq_num 217) 2024-03-27T09:36:45.950 INFO:tasks.rados.rados.0.smithi170.stdout:423: finishing write tid 1 to smithi17031214-12 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:421: finishing write tid 2 to smithi17031214-13 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: finishing write tid 2 to smithi17031214-12 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: finishing write tid 3 to smithi17031214-12 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:421: finishing write tid 3 to smithi17031214-13 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: finishing write tid 4 to smithi17031214-12 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: finishing write tid 5 to smithi17031214-12 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: oid 12 updating version 0 to 102 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: oid 12 version 102 is already newer than 101 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: oid 12 updating version 102 to 104 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: oid 12 version 104 is already newer than 103 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 104 (ObjNum 218 snap 47 seq_num 218) dirty exists 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:423: left oid 12 (ObjNum 218 snap 47 seq_num 218) 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:421: finishing write tid 4 to smithi17031214-13 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:421: finishing write tid 5 to smithi17031214-13 2024-03-27T09:36:45.951 INFO:tasks.rados.rados.0.smithi170.stdout:421: oid 13 updating version 0 to 89 2024-03-27T09:36:45.952 INFO:tasks.rados.rados.0.smithi170.stdout:421: oid 13 version 89 is already newer than 88 2024-03-27T09:36:45.952 INFO:tasks.rados.rados.0.smithi170.stdout:421: oid 13 updating version 89 to 90 2024-03-27T09:36:45.952 INFO:tasks.rados.rados.0.smithi170.stdout:421: oid 13 updating version 90 to 91 2024-03-27T09:36:45.952 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 91 (ObjNum 216 snap 47 seq_num 216) dirty exists 2024-03-27T09:36:45.952 INFO:tasks.rados.rados.0.smithi170.stdout:421: left oid 13 (ObjNum 216 snap 47 seq_num 216) 2024-03-27T09:36:45.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:46.066 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:46.064+0000 7f9ef7e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:46.069 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:46.069+0000 7f9ef7e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:46.093 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:46.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:46.949 INFO:tasks.rados.rados.0.smithi170.stdout:421: done (3 left) 2024-03-27T09:36:46.949 INFO:tasks.rados.rados.0.smithi170.stdout:422: done (2 left) 2024-03-27T09:36:46.949 INFO:tasks.rados.rados.0.smithi170.stdout:423: done (1 left) 2024-03-27T09:36:46.950 INFO:tasks.rados.rados.0.smithi170.stdout:425: done (0 left) 2024-03-27T09:36:46.950 INFO:tasks.rados.rados.0.smithi170.stdout:426: snap_create 2024-03-27T09:36:47.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:47.490 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:47.493 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:47.950 INFO:tasks.rados.rados.0.smithi170.stdout:426: done (0 left) 2024-03-27T09:36:47.950 INFO:tasks.rados.rados.0.smithi170.stdout:427: write (excl) oid 14 current snap is 49 2024-03-27T09:36:47.950 INFO:tasks.rados.rados.0.smithi170.stdout:427: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-03-27T09:36:47.956 INFO:tasks.rados.rados.0.smithi170.stdout:427: writing smithi17031214-14 from 480733 to 1018148 tid 1 2024-03-27T09:36:47.964 INFO:tasks.rados.rados.0.smithi170.stdout:427: writing smithi17031214-14 from 1782380 to 2478782 tid 2 2024-03-27T09:36:47.971 INFO:tasks.rados.rados.0.smithi170.stdout:427: writing smithi17031214-14 from 3031400 to 3568559 tid 3 2024-03-27T09:36:47.971 INFO:tasks.rados.rados.0.smithi170.stdout:428: write (excl) oid 17 current snap is 49 2024-03-27T09:36:47.971 INFO:tasks.rados.rados.0.smithi170.stdout:428: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-03-27T09:36:47.978 INFO:tasks.rados.rados.0.smithi170.stdout:428: writing smithi17031214-17 from 497540 to 1110204 tid 1 2024-03-27T09:36:47.985 INFO:tasks.rados.rados.0.smithi170.stdout:428: writing smithi17031214-17 from 1840862 to 2280922 tid 2 2024-03-27T09:36:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:428: writing smithi17031214-17 from 3058823 to 3523607 tid 3 2024-03-27T09:36:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:429: write oid 24 current snap is 49 2024-03-27T09:36:47.995 INFO:tasks.rados.rados.0.smithi170.stdout:429: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-03-27T09:36:48.001 INFO:tasks.rados.rados.0.smithi170.stdout:429: writing smithi17031214-24 from 514347 to 918613 tid 1 2024-03-27T09:36:48.011 INFO:tasks.rados.rados.0.smithi170.stdout:429: writing smithi17031214-24 from 1499344 to 2199415 tid 2 2024-03-27T09:36:48.021 INFO:tasks.rados.rados.0.smithi170.stdout:429: writing smithi17031214-24 from 2686246 to 3362302 tid 3 2024-03-27T09:36:48.021 INFO:tasks.rados.rados.0.smithi170.stdout:430: snap_remove snap 41 2024-03-27T09:36:48.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:48.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:48.260+0000 7f8ed6c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:48.266 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:48.265+0000 7f8ed6c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:48.290 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:48.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:430: done (3 left) 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:431: snap_create 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:429: finishing write tid 1 to smithi17031214-24 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:427: finishing write tid 1 to smithi17031214-14 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:428: finishing write tid 1 to smithi17031214-17 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:429: finishing write tid 2 to smithi17031214-24 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:429: finishing write tid 3 to smithi17031214-24 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:427: finishing write tid 2 to smithi17031214-14 2024-03-27T09:36:49.005 INFO:tasks.rados.rados.0.smithi170.stdout:428: finishing write tid 2 to smithi17031214-17 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: finishing write tid 4 to smithi17031214-24 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: finishing write tid 5 to smithi17031214-24 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: oid 24 updating version 0 to 42 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: oid 24 updating version 42 to 44 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: oid 24 updating version 44 to 45 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: oid 24 version 45 is already newer than 44 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 45 (ObjNum 221 snap 49 seq_num 221) dirty exists 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:429: left oid 24 (ObjNum 221 snap 49 seq_num 221) 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:428: finishing write tid 3 to smithi17031214-17 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:428: finishing write tid 4 to smithi17031214-17 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:428: finishing write tid 5 to smithi17031214-17 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:428: oid 17 updating version 0 to 131 2024-03-27T09:36:49.006 INFO:tasks.rados.rados.0.smithi170.stdout:428: oid 17 version 131 is already newer than 128 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:428: oid 17 version 131 is already newer than 130 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:428: oid 17 version 131 is already newer than 130 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 131 (ObjNum 220 snap 49 seq_num 220) dirty exists 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:428: left oid 17 (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: finishing write tid 3 to smithi17031214-14 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: finishing write tid 4 to smithi17031214-14 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: finishing write tid 5 to smithi17031214-14 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: oid 14 updating version 0 to 99 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: oid 14 updating version 99 to 100 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: oid 14 version 100 is already newer than 93 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: oid 14 version 100 is already newer than 94 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 100 (ObjNum 219 snap 49 seq_num 219) dirty exists 2024-03-27T09:36:49.007 INFO:tasks.rados.rados.0.smithi170.stdout:427: left oid 14 (ObjNum 219 snap 49 seq_num 219) 2024-03-27T09:36:50.008 INFO:tasks.rados.rados.0.smithi170.stdout:427: done (3 left) 2024-03-27T09:36:50.008 INFO:tasks.rados.rados.0.smithi170.stdout:428: done (2 left) 2024-03-27T09:36:50.008 INFO:tasks.rados.rados.0.smithi170.stdout:429: done (1 left) 2024-03-27T09:36:50.008 INFO:tasks.rados.rados.0.smithi170.stdout:431: done (0 left) 2024-03-27T09:36:50.008 INFO:tasks.rados.rados.0.smithi170.stdout:432: delete oid 12 current snap is 50 2024-03-27T09:36:50.090 INFO:tasks.rados.rados.0.smithi170.stdout:432: done (0 left) 2024-03-27T09:36:50.090 INFO:tasks.rados.rados.0.smithi170.stdout:433: write (excl) oid 43 current snap is 50 2024-03-27T09:36:50.090 INFO:tasks.rados.rados.0.smithi170.stdout:433: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-03-27T09:36:50.096 INFO:tasks.rados.rados.0.smithi170.stdout:433: writing smithi17031214-43 from 547961 to 1102725 tid 1 2024-03-27T09:36:50.102 INFO:tasks.rados.rados.0.smithi170.stdout:433: writing smithi17031214-43 from 1616308 to 2320048 tid 2 2024-03-27T09:36:50.108 INFO:tasks.rados.rados.0.smithi170.stdout:433: writing smithi17031214-43 from 2741092 to 3439692 tid 3 2024-03-27T09:36:50.109 INFO:tasks.rados.rados.0.smithi170.stdout:434: snap_create 2024-03-27T09:36:50.119 INFO:tasks.rados.rados.0.smithi170.stdout:433: finishing write tid 1 to smithi17031214-43 2024-03-27T09:36:50.141 INFO:tasks.rados.rados.0.smithi170.stdout:433: finishing write tid 2 to smithi17031214-43 2024-03-27T09:36:50.156 INFO:tasks.rados.rados.0.smithi170.stdout:433: finishing write tid 3 to smithi17031214-43 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: finishing write tid 4 to smithi17031214-43 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: finishing write tid 5 to smithi17031214-43 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: oid 43 updating version 0 to 142 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: oid 43 updating version 142 to 144 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: oid 43 version 144 is already newer than 143 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: oid 43 version 144 is already newer than 141 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 144 (ObjNum 223 snap 50 seq_num 223) dirty exists 2024-03-27T09:36:50.167 INFO:tasks.rados.rados.0.smithi170.stdout:433: left oid 43 (ObjNum 223 snap 50 seq_num 223) 2024-03-27T09:36:50.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:50.457 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:50.454+0000 7ff7041ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:50.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:50.460+0000 7ff702f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:50.485 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:50.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:50.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~61s 2024-03-27T09:36:51.010 INFO:tasks.rados.rados.0.smithi170.stdout:433: done (1 left) 2024-03-27T09:36:51.010 INFO:tasks.rados.rados.0.smithi170.stdout:434: done (0 left) 2024-03-27T09:36:51.010 INFO:tasks.rados.rados.0.smithi170.stdout:435: snap_create 2024-03-27T09:36:52.015 INFO:tasks.rados.rados.0.smithi170.stdout:435: done (0 left) 2024-03-27T09:36:52.015 INFO:tasks.rados.rados.0.smithi170.stdout:436: write oid 45 current snap is 52 2024-03-27T09:36:52.015 INFO:tasks.rados.rados.0.smithi170.stdout:436: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-03-27T09:36:52.021 INFO:tasks.rados.rados.0.smithi170.stdout:436: writing smithi17031214-45 from 564768 to 1194781 tid 1 2024-03-27T09:36:52.026 INFO:tasks.rados.rados.0.smithi170.stdout:436: writing smithi17031214-45 from 1674790 to 2122188 tid 2 2024-03-27T09:36:52.033 INFO:tasks.rados.rados.0.smithi170.stdout:436: writing smithi17031214-45 from 2768515 to 3394740 tid 3 2024-03-27T09:36:52.033 INFO:tasks.rados.rados.0.smithi170.stdout:437: snap_create 2024-03-27T09:36:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:436: finishing write tid 1 to smithi17031214-45 2024-03-27T09:36:52.126 INFO:tasks.rados.rados.0.smithi170.stdout:436: finishing write tid 2 to smithi17031214-45 2024-03-27T09:36:52.128 INFO:tasks.rados.rados.0.smithi170.stdout:436: finishing write tid 3 to smithi17031214-45 2024-03-27T09:36:52.139 INFO:tasks.rados.rados.0.smithi170.stdout:436: finishing write tid 4 to smithi17031214-45 2024-03-27T09:36:52.140 INFO:tasks.rados.rados.0.smithi170.stdout:436: finishing write tid 5 to smithi17031214-45 2024-03-27T09:36:52.140 INFO:tasks.rados.rados.0.smithi170.stdout:436: oid 45 updating version 0 to 146 2024-03-27T09:36:52.141 INFO:tasks.rados.rados.0.smithi170.stdout:436: oid 45 updating version 146 to 148 2024-03-27T09:36:52.141 INFO:tasks.rados.rados.0.smithi170.stdout:436: oid 45 version 148 is already newer than 148 2024-03-27T09:36:52.141 INFO:tasks.rados.rados.0.smithi170.stdout:436: oid 45 updating version 148 to 149 2024-03-27T09:36:52.141 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 149 (ObjNum 224 snap 52 seq_num 224) dirty exists 2024-03-27T09:36:52.141 INFO:tasks.rados.rados.0.smithi170.stdout:436: left oid 45 (ObjNum 224 snap 52 seq_num 224) 2024-03-27T09:36:52.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:52.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:52.608 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:52.611 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:52.652+0000 7f811bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:52.658+0000 7f811bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:52.682 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:52.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:53.017 INFO:tasks.rados.rados.0.smithi170.stdout:436: done (1 left) 2024-03-27T09:36:53.017 INFO:tasks.rados.rados.0.smithi170.stdout:437: done (0 left) 2024-03-27T09:36:53.017 INFO:tasks.rados.rados.0.smithi170.stdout:438: snap_create 2024-03-27T09:36:54.019 INFO:tasks.rados.rados.0.smithi170.stdout:438: done (0 left) 2024-03-27T09:36:54.019 INFO:tasks.rados.rados.0.smithi170.stdout:439: delete oid 13 current snap is 54 2024-03-27T09:36:54.120 INFO:tasks.rados.rados.0.smithi170.stdout:439: done (0 left) 2024-03-27T09:36:54.120 INFO:tasks.rados.rados.0.smithi170.stdout:440: write (excl) oid 36 current snap is 54 2024-03-27T09:36:54.120 INFO:tasks.rados.rados.0.smithi170.stdout:440: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-03-27T09:36:54.127 INFO:tasks.rados.rados.0.smithi170.stdout:440: writing smithi17031214-36 from 598382 to 1378893 tid 1 2024-03-27T09:36:54.133 INFO:tasks.rados.rados.0.smithi170.stdout:440: writing smithi17031214-36 from 2075401 to 2526468 tid 2 2024-03-27T09:36:54.140 INFO:tasks.rados.rados.0.smithi170.stdout:440: writing smithi17031214-36 from 3107008 to 3755777 tid 3 2024-03-27T09:36:54.140 INFO:tasks.rados.rados.0.smithi170.stdout:441: snap_create 2024-03-27T09:36:54.229 INFO:tasks.rados.rados.0.smithi170.stdout:440: finishing write tid 1 to smithi17031214-36 2024-03-27T09:36:54.242 INFO:tasks.rados.rados.0.smithi170.stdout:440: finishing write tid 2 to smithi17031214-36 2024-03-27T09:36:54.259 INFO:tasks.rados.rados.0.smithi170.stdout:440: finishing write tid 3 to smithi17031214-36 2024-03-27T09:36:54.266 INFO:tasks.rados.rados.0.smithi170.stdout:440: finishing write tid 4 to smithi17031214-36 2024-03-27T09:36:54.267 INFO:tasks.rados.rados.0.smithi170.stdout:440: finishing write tid 5 to smithi17031214-36 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:440: oid 36 updating version 0 to 136 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:440: oid 36 updating version 136 to 138 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:440: oid 36 version 138 is already newer than 135 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:440: oid 36 version 138 is already newer than 137 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 138 (ObjNum 226 snap 54 seq_num 226) dirty exists 2024-03-27T09:36:54.268 INFO:tasks.rados.rados.0.smithi170.stdout:440: left oid 36 (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:36:54.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:54.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:54.837+0000 7f4d56354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:54.843 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:54.842+0000 7f4d56354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:54.866 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:54.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:440: done (1 left) 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:441: done (0 left) 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:442: read oid 48 snap -1 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:443: read oid 43 snap 42 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:443: expect deleted 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:444: write oid 50 current snap is 55 2024-03-27T09:36:55.025 INFO:tasks.rados.rados.0.smithi170.stdout:444: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-03-27T09:36:55.031 INFO:tasks.rados.rados.0.smithi170.stdout:444: writing smithi17031214-50 from 615189 to 1070949 tid 1 2024-03-27T09:36:55.039 INFO:tasks.rados.rados.0.smithi170.stdout:444: writing smithi17031214-50 from 1733883 to 2444961 tid 2 2024-03-27T09:36:55.048 INFO:tasks.rados.rados.0.smithi170.stdout:444: writing smithi17031214-50 from 3134431 to 3594472 tid 3 2024-03-27T09:36:55.048 INFO:tasks.rados.rados.0.smithi170.stdout:445: read oid 1 snap -1 2024-03-27T09:36:55.048 INFO:tasks.rados.rados.0.smithi170.stdout:445: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:36:55.048 INFO:tasks.rados.rados.0.smithi170.stdout:446: write oid 29 current snap is 55 2024-03-27T09:36:55.048 INFO:tasks.rados.rados.0.smithi170.stdout:446: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-03-27T09:36:55.056 INFO:tasks.rados.rados.0.smithi170.stdout:446: writing smithi17031214-29 from 631996 to 1163005 tid 1 2024-03-27T09:36:55.065 INFO:tasks.rados.rados.0.smithi170.stdout:446: writing smithi17031214-29 from 1792365 to 2247101 tid 2 2024-03-27T09:36:55.072 INFO:tasks.rados.rados.0.smithi170.stdout:446: writing smithi17031214-29 from 2761854 to 3433167 tid 3 2024-03-27T09:36:55.072 INFO:tasks.rados.rados.0.smithi170.stdout:447: snap_remove snap 49 2024-03-27T09:36:55.951 INFO:tasks.rados.rados.0.smithi170.stdout:447: done (5 left) 2024-03-27T09:36:55.951 INFO:tasks.rados.rados.0.smithi170.stdout:448: delete oid 12 current snap is 55 2024-03-27T09:36:55.952 INFO:tasks.rados.rados.0.smithi170.stdout:445: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:36:56.041 INFO:tasks.rados.rados.0.smithi170.stdout:442: expect (ObjNum 217 snap 47 seq_num 217) 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:446: finishing write tid 1 to smithi17031214-29 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:444: finishing write tid 1 to smithi17031214-50 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:446: finishing write tid 2 to smithi17031214-29 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:442: done (5 left) 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:443: done (4 left) 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:445: done (3 left) 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:448: done (2 left) 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:449: read oid 10 snap 47 2024-03-27T09:36:56.270 INFO:tasks.rados.rados.0.smithi170.stdout:449: expect (ObjNum 147 snap 20 seq_num 147) 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:450: read oid 18 snap -1 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:450: expect (ObjNum 185 snap 34 seq_num 185) 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:451: write (excl) oid 27 current snap is 55 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:444: finishing write tid 2 to smithi17031214-50 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: finishing write tid 3 to smithi17031214-29 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: finishing write tid 4 to smithi17031214-29 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:444: finishing write tid 3 to smithi17031214-50 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: finishing write tid 5 to smithi17031214-29 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: oid 29 updating version 0 to 115 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: oid 29 updating version 115 to 117 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: oid 29 updating version 117 to 118 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: oid 29 version 118 is already newer than 117 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 118 (ObjNum 228 snap 55 seq_num 228) dirty exists 2024-03-27T09:36:56.271 INFO:tasks.rados.rados.0.smithi170.stdout:446: left oid 29 (ObjNum 228 snap 55 seq_num 228) 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: finishing write tid 4 to smithi17031214-50 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: finishing write tid 5 to smithi17031214-50 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: oid 50 updating version 0 to 143 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: oid 50 version 143 is already newer than 142 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: oid 50 version 143 is already newer than 140 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: oid 50 version 143 is already newer than 141 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 143 (ObjNum 227 snap 55 seq_num 227) dirty exists 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:444: left oid 50 (ObjNum 227 snap 55 seq_num 227) 2024-03-27T09:36:56.272 INFO:tasks.rados.rados.0.smithi170.stdout:451: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-03-27T09:36:56.281 INFO:tasks.rados.rados.0.smithi170.stdout:451: writing smithi17031214-27 from 665610 to 1463470 tid 1 2024-03-27T09:36:56.288 INFO:tasks.rados.rados.0.smithi170.stdout:451: writing smithi17031214-27 from 1909329 to 2367734 tid 2 2024-03-27T09:36:56.294 INFO:tasks.rados.rados.0.smithi170.stdout:451: writing smithi17031214-27 from 2816700 to 3226910 tid 3 2024-03-27T09:36:56.294 INFO:tasks.rados.rados.0.smithi170.stdout:444: done (4 left) 2024-03-27T09:36:56.294 INFO:tasks.rados.rados.0.smithi170.stdout:446: done (3 left) 2024-03-27T09:36:56.295 INFO:tasks.rados.rados.0.smithi170.stdout:452: read oid 44 snap 47 2024-03-27T09:36:56.295 INFO:tasks.rados.rados.0.smithi170.stdout:452: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:36:56.295 INFO:tasks.rados.rados.0.smithi170.stdout:453: write (excl) oid 39 current snap is 55 2024-03-27T09:36:56.295 INFO:tasks.rados.rados.0.smithi170.stdout:453: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-03-27T09:36:56.302 INFO:tasks.rados.rados.0.smithi170.stdout:453: writing smithi17031214-39 from 682417 to 1155526 tid 1 2024-03-27T09:36:56.310 INFO:tasks.rados.rados.0.smithi170.stdout:453: writing smithi17031214-39 from 1567811 to 2169874 tid 2 2024-03-27T09:36:56.318 INFO:tasks.rados.rados.0.smithi170.stdout:453: writing smithi17031214-39 from 2844123 to 3465605 tid 3 2024-03-27T09:36:56.318 INFO:tasks.rados.rados.0.smithi170.stdout:454: write oid 14 current snap is 55 2024-03-27T09:36:56.318 INFO:tasks.rados.rados.0.smithi170.stdout:454: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-03-27T09:36:56.325 INFO:tasks.rados.rados.0.smithi170.stdout:454: writing smithi17031214-14 from 699224 to 1247582 tid 1 2024-03-27T09:36:56.332 INFO:tasks.rados.rados.0.smithi170.stdout:454: writing smithi17031214-14 from 2026293 to 2488367 tid 2 2024-03-27T09:36:56.341 INFO:tasks.rados.rados.0.smithi170.stdout:454: writing smithi17031214-14 from 3271546 to 3704300 tid 3 2024-03-27T09:36:56.341 INFO:tasks.rados.rados.0.smithi170.stdout:455: snap_create 2024-03-27T09:36:56.341 INFO:tasks.rados.rados.0.smithi170.stdout:449: expect (ObjNum 147 snap 20 seq_num 147) 2024-03-27T09:36:56.450 INFO:tasks.rados.rados.0.smithi170.stdout:450: expect (ObjNum 185 snap 34 seq_num 185) 2024-03-27T09:36:56.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~66s 2024-03-27T09:36:56.684 INFO:tasks.rados.rados.0.smithi170.stdout:451: finishing write tid 1 to smithi17031214-27 2024-03-27T09:36:56.685 INFO:tasks.rados.rados.0.smithi170.stdout:451: finishing write tid 2 to smithi17031214-27 2024-03-27T09:36:56.685 INFO:tasks.rados.rados.0.smithi170.stdout:452: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-27T09:36:56.908 INFO:tasks.rados.rados.0.smithi170.stdout:451: finishing write tid 3 to smithi17031214-27 2024-03-27T09:36:56.908 INFO:tasks.rados.rados.0.smithi170.stdout:454: finishing write tid 1 to smithi17031214-14 2024-03-27T09:36:56.908 INFO:tasks.rados.rados.0.smithi170.stdout:451: finishing write tid 4 to smithi17031214-27 2024-03-27T09:36:56.908 INFO:tasks.rados.rados.0.smithi170.stdout:453: finishing write tid 1 to smithi17031214-39 2024-03-27T09:36:56.908 INFO:tasks.rados.rados.0.smithi170.stdout:451: finishing write tid 5 to smithi17031214-27 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:451: oid 27 updating version 0 to 155 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:451: oid 27 version 155 is already newer than 153 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:451: oid 27 version 155 is already newer than 152 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:451: oid 27 version 155 is already newer than 154 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 155 (ObjNum 230 snap 55 seq_num 230) dirty exists 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:451: left oid 27 (ObjNum 230 snap 55 seq_num 230) 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:454: finishing write tid 2 to smithi17031214-14 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: finishing write tid 2 to smithi17031214-39 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: finishing write tid 3 to smithi17031214-39 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: finishing write tid 4 to smithi17031214-39 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:454: finishing write tid 3 to smithi17031214-14 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: finishing write tid 5 to smithi17031214-39 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: oid 39 updating version 0 to 47 2024-03-27T09:36:56.909 INFO:tasks.rados.rados.0.smithi170.stdout:453: oid 39 updating version 47 to 50 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:453: oid 39 version 50 is already newer than 49 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:453: oid 39 version 50 is already newer than 49 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 50 (ObjNum 231 snap 55 seq_num 231) dirty exists 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:453: left oid 39 (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: finishing write tid 4 to smithi17031214-14 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: finishing write tid 5 to smithi17031214-14 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: oid 14 updating version 0 to 113 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: oid 14 version 113 is already newer than 112 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: oid 14 version 113 is already newer than 110 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: oid 14 version 113 is already newer than 111 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 113 (ObjNum 232 snap 55 seq_num 232) dirty exists 2024-03-27T09:36:56.910 INFO:tasks.rados.rados.0.smithi170.stdout:454: left oid 14 (ObjNum 232 snap 55 seq_num 232) 2024-03-27T09:36:56.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:36:57.026 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:57.023+0000 7fcf72aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:57.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:57.029+0000 7fcf72aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:449: done (6 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:450: done (5 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:451: done (4 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:452: done (3 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:453: done (2 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:454: done (1 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:455: done (0 left) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:456: read oid 33 snap 55 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2024-03-27T09:36:57.033 INFO:tasks.rados.rados.0.smithi170.stdout:457: snap_create 2024-03-27T09:36:57.074 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:57.098 INFO:tasks.rados.rados.0.smithi170.stdout:456: expect (ObjNum 200 snap 41 seq_num 200) 2024-03-27T09:36:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:57.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:36:57.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:36:57.713 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:36:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:36:57.716 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:36:58.035 INFO:tasks.rados.rados.0.smithi170.stdout:456: done (1 left) 2024-03-27T09:36:58.035 INFO:tasks.rados.rados.0.smithi170.stdout:457: done (0 left) 2024-03-27T09:36:58.035 INFO:tasks.rados.rados.0.smithi170.stdout:458: write (excl) oid 45 current snap is 57 2024-03-27T09:36:58.035 INFO:tasks.rados.rados.0.smithi170.stdout:458: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-03-27T09:36:58.043 INFO:tasks.rados.rados.0.smithi170.stdout:458: writing smithi17031214-45 from 716031 to 1339638 tid 1 2024-03-27T09:36:58.053 INFO:tasks.rados.rados.0.smithi170.stdout:458: writing smithi17031214-45 from 2084775 to 2690507 tid 2 2024-03-27T09:36:58.063 INFO:tasks.rados.rados.0.smithi170.stdout:458: writing smithi17031214-45 from 3298969 to 3916031 tid 3 2024-03-27T09:36:58.063 INFO:tasks.rados.rados.0.smithi170.stdout:459: snap_create 2024-03-27T09:36:58.118 INFO:tasks.rados.rados.0.smithi170.stdout:458: finishing write tid 1 to smithi17031214-45 2024-03-27T09:36:58.143 INFO:tasks.rados.rados.0.smithi170.stdout:458: finishing write tid 2 to smithi17031214-45 2024-03-27T09:36:58.148 INFO:tasks.rados.rados.0.smithi170.stdout:458: finishing write tid 3 to smithi17031214-45 2024-03-27T09:36:58.156 INFO:tasks.rados.rados.0.smithi170.stdout:458: finishing write tid 4 to smithi17031214-45 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: finishing write tid 5 to smithi17031214-45 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: oid 45 updating version 0 to 159 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: oid 45 version 159 is already newer than 159 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: oid 45 version 159 is already newer than 157 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: oid 45 updating version 159 to 160 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 160 (ObjNum 233 snap 57 seq_num 233) dirty exists 2024-03-27T09:36:58.158 INFO:tasks.rados.rados.0.smithi170.stdout:458: left oid 45 (ObjNum 233 snap 57 seq_num 233) 2024-03-27T09:36:59.039 INFO:tasks.rados.rados.0.smithi170.stdout:458: done (1 left) 2024-03-27T09:36:59.039 INFO:tasks.rados.rados.0.smithi170.stdout:459: done (0 left) 2024-03-27T09:36:59.039 INFO:tasks.rados.rados.0.smithi170.stdout:460: read oid 35 snap 52 2024-03-27T09:36:59.040 INFO:tasks.rados.rados.0.smithi170.stdout:460: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:36:59.040 INFO:tasks.rados.rados.0.smithi170.stdout:461: read oid 2 snap -1 2024-03-27T09:36:59.040 INFO:tasks.rados.rados.0.smithi170.stdout:461: expect deleted 2024-03-27T09:36:59.040 INFO:tasks.rados.rados.0.smithi170.stdout:462: snap_remove snap 48 2024-03-27T09:36:59.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:36:59.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:59.260+0000 7f9d69bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:59.266 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:36:59.265+0000 7f9d69bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:36:59.289 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:36:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:36:59.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:462: done (2 left) 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:463: read oid 9 snap -1 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:463: expect (ObjNum 170 snap 26 seq_num 170) 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:464: read oid 40 snap -1 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:465: write (excl) oid 24 current snap is 58 2024-03-27T09:37:00.043 INFO:tasks.rados.rados.0.smithi170.stdout:465: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-03-27T09:37:00.051 INFO:tasks.rados.rados.0.smithi170.stdout:465: writing smithi17031214-24 from 732838 to 1431694 tid 1 2024-03-27T09:37:00.059 INFO:tasks.rados.rados.0.smithi170.stdout:465: writing smithi17031214-24 from 2026904 to 2492647 tid 2 2024-03-27T09:37:00.069 INFO:tasks.rados.rados.0.smithi170.stdout:465: writing smithi17031214-24 from 3210039 to 3781690 tid 3 2024-03-27T09:37:00.069 INFO:tasks.rados.rados.0.smithi170.stdout:466: read oid 17 snap -1 2024-03-27T09:37:00.069 INFO:tasks.rados.rados.0.smithi170.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:00.069 INFO:tasks.rados.rados.0.smithi170.stdout:467: snap_remove snap 32 2024-03-27T09:37:01.046 INFO:tasks.rados.rados.0.smithi170.stdout:467: done (6 left) 2024-03-27T09:37:01.046 INFO:tasks.rados.rados.0.smithi170.stdout:468: delete oid 20 current snap is 58 2024-03-27T09:37:01.046 INFO:tasks.rados.rados.0.smithi170.stdout:460: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:37:01.292 INFO:tasks.rados.rados.0.smithi170.stdout:463: expect (ObjNum 170 snap 26 seq_num 170) 2024-03-27T09:37:01.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:01.446 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:01.444+0000 7ff2d9b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:01.450 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:01.450+0000 7ff2d9b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:01.475 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:01.484 INFO:tasks.rados.rados.0.smithi170.stdout:464: expect (ObjNum 122 snap 14 seq_num 122) 2024-03-27T09:37:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:01.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:01.607 INFO:tasks.rados.rados.0.smithi170.stdout:466: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:01.927 INFO:tasks.rados.rados.0.smithi170.stdout:465: finishing write tid 1 to smithi17031214-24 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: finishing write tid 2 to smithi17031214-24 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: finishing write tid 3 to smithi17031214-24 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: finishing write tid 4 to smithi17031214-24 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: finishing write tid 5 to smithi17031214-24 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: oid 24 updating version 0 to 51 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: oid 24 updating version 51 to 53 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: oid 24 version 53 is already newer than 53 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: oid 24 updating version 53 to 54 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 54 (ObjNum 234 snap 58 seq_num 234) dirty exists 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: left oid 24 (ObjNum 234 snap 58 seq_num 234) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:460: done (6 left) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:461: done (5 left) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:463: done (4 left) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:464: done (3 left) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:465: done (2 left) 2024-03-27T09:37:01.928 INFO:tasks.rados.rados.0.smithi170.stdout:466: done (1 left) 2024-03-27T09:37:01.929 INFO:tasks.rados.rados.0.smithi170.stdout:468: done (0 left) 2024-03-27T09:37:01.929 INFO:tasks.rados.rados.0.smithi170.stdout:469: read oid 15 snap 55 2024-03-27T09:37:01.929 INFO:tasks.rados.rados.0.smithi170.stdout:469: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:01.929 INFO:tasks.rados.rados.0.smithi170.stdout:470: snap_create 2024-03-27T09:37:01.942 INFO:tasks.rados.rados.0.smithi170.stdout:469: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:01.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~72s 2024-03-27T09:37:02.067 INFO:tasks.rados.rados.0.smithi170.stdout:469: done (1 left) 2024-03-27T09:37:02.067 INFO:tasks.rados.rados.0.smithi170.stdout:470: done (0 left) 2024-03-27T09:37:02.067 INFO:tasks.rados.rados.0.smithi170.stdout:471: delete oid 12 current snap is 59 2024-03-27T09:37:02.109 INFO:tasks.rados.rados.0.smithi170.stdout:471: done (0 left) 2024-03-27T09:37:02.109 INFO:tasks.rados.rados.0.smithi170.stdout:472: snap_remove snap 52 2024-03-27T09:37:02.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:02.820 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:02.823 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:03.051 INFO:tasks.rados.rados.0.smithi170.stdout:472: done (0 left) 2024-03-27T09:37:03.052 INFO:tasks.rados.rados.0.smithi170.stdout:473: snap_create 2024-03-27T09:37:03.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:03.638 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:03.635+0000 7f42d2fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:03.641 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:03.641+0000 7f42d1d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:03.665 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:03.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:04.055 INFO:tasks.rados.rados.0.smithi170.stdout:473: done (0 left) 2024-03-27T09:37:04.055 INFO:tasks.rados.rados.0.smithi170.stdout:474: snap_remove snap 47 2024-03-27T09:37:05.057 INFO:tasks.rados.rados.0.smithi170.stdout:474: done (0 left) 2024-03-27T09:37:05.057 INFO:tasks.rados.rados.0.smithi170.stdout:475: read oid 6 snap -1 2024-03-27T09:37:05.057 INFO:tasks.rados.rados.0.smithi170.stdout:475: expect deleted 2024-03-27T09:37:05.057 INFO:tasks.rados.rados.0.smithi170.stdout:476: snap_remove snap 55 2024-03-27T09:37:05.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:05.834 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:05.831+0000 7f3784c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:05.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:05.836+0000 7f3784c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:05.862 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:05.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:476: done (1 left) 2024-03-27T09:37:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:477: read oid 25 snap 46 2024-03-27T09:37:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:477: expect (ObjNum 164 snap 24 seq_num 164) 2024-03-27T09:37:05.952 INFO:tasks.rados.rados.0.smithi170.stdout:478: delete oid 2 current snap is 60 2024-03-27T09:37:06.042 INFO:tasks.rados.rados.0.smithi170.stdout:477: expect (ObjNum 164 snap 24 seq_num 164) 2024-03-27T09:37:06.234 INFO:tasks.rados.rados.0.smithi170.stdout:475: done (2 left) 2024-03-27T09:37:06.235 INFO:tasks.rados.rados.0.smithi170.stdout:477: done (1 left) 2024-03-27T09:37:06.235 INFO:tasks.rados.rados.0.smithi170.stdout:478: done (0 left) 2024-03-27T09:37:06.235 INFO:tasks.rados.rados.0.smithi170.stdout:479: read oid 16 snap -1 2024-03-27T09:37:06.235 INFO:tasks.rados.rados.0.smithi170.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:06.235 INFO:tasks.rados.rados.0.smithi170.stdout:480: snap_create 2024-03-27T09:37:06.257 INFO:tasks.rados.rados.0.smithi170.stdout:479: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:07.065 INFO:tasks.rados.rados.0.smithi170.stdout:479: done (1 left) 2024-03-27T09:37:07.065 INFO:tasks.rados.rados.0.smithi170.stdout:480: done (0 left) 2024-03-27T09:37:07.065 INFO:tasks.rados.rados.0.smithi170.stdout:481: snap_create 2024-03-27T09:37:07.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2024-03-27T09:37:07.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:07.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:07.935 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:07.938 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:08.030 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:08.028+0000 7f461659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:08.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:08.033+0000 7f461659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:08.057 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:08.069 INFO:tasks.rados.rados.0.smithi170.stdout:481: done (0 left) 2024-03-27T09:37:08.070 INFO:tasks.rados.rados.0.smithi170.stdout:482: snap_remove snap 61 2024-03-27T09:37:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:08.113 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:09.071 INFO:tasks.rados.rados.0.smithi170.stdout:482: done (0 left) 2024-03-27T09:37:09.071 INFO:tasks.rados.rados.0.smithi170.stdout:483: delete oid 48 current snap is 62 2024-03-27T09:37:09.158 INFO:tasks.rados.rados.0.smithi170.stdout:483: done (0 left) 2024-03-27T09:37:09.158 INFO:tasks.rados.rados.0.smithi170.stdout:484: read oid 46 snap -1 2024-03-27T09:37:09.158 INFO:tasks.rados.rados.0.smithi170.stdout:484: expect deleted 2024-03-27T09:37:09.158 INFO:tasks.rados.rados.0.smithi170.stdout:485: write oid 40 current snap is 62 2024-03-27T09:37:09.158 INFO:tasks.rados.rados.0.smithi170.stdout:485: seq_num 239 ranges {416873=791454,1919314=97559} 2024-03-27T09:37:09.167 INFO:tasks.rados.rados.0.smithi170.stdout:485: writing smithi17031214-40 from 416873 to 1208327 tid 1 2024-03-27T09:37:09.172 INFO:tasks.rados.rados.0.smithi170.stdout:485: writing smithi17031214-40 from 1919314 to 2016873 tid 2 2024-03-27T09:37:09.173 INFO:tasks.rados.rados.0.smithi170.stdout:486: snap_create 2024-03-27T09:37:09.218 INFO:tasks.rados.rados.0.smithi170.stdout:485: finishing write tid 1 to smithi17031214-40 2024-03-27T09:37:09.232 INFO:tasks.rados.rados.0.smithi170.stdout:485: finishing write tid 2 to smithi17031214-40 2024-03-27T09:37:09.238 INFO:tasks.rados.rados.0.smithi170.stdout:485: finishing write tid 3 to smithi17031214-40 2024-03-27T09:37:09.239 INFO:tasks.rados.rados.0.smithi170.stdout:485: finishing write tid 4 to smithi17031214-40 2024-03-27T09:37:09.240 INFO:tasks.rados.rados.0.smithi170.stdout:485: oid 40 updating version 0 to 134 2024-03-27T09:37:09.240 INFO:tasks.rados.rados.0.smithi170.stdout:485: oid 40 version 134 is already newer than 134 2024-03-27T09:37:09.240 INFO:tasks.rados.rados.0.smithi170.stdout:485: oid 40 version 134 is already newer than 132 2024-03-27T09:37:09.240 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 134 (ObjNum 239 snap 62 seq_num 239) dirty exists 2024-03-27T09:37:09.240 INFO:tasks.rados.rados.0.smithi170.stdout:485: left oid 40 (ObjNum 239 snap 62 seq_num 239) 2024-03-27T09:37:10.078 INFO:tasks.rados.rados.0.smithi170.stdout:484: done (2 left) 2024-03-27T09:37:10.078 INFO:tasks.rados.rados.0.smithi170.stdout:485: done (1 left) 2024-03-27T09:37:10.078 INFO:tasks.rados.rados.0.smithi170.stdout:486: done (0 left) 2024-03-27T09:37:10.078 INFO:tasks.rados.rados.0.smithi170.stdout:487: snap_create 2024-03-27T09:37:10.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:10.233 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:10.230+0000 7f066902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:10.237 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:10.236+0000 7f066902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:10.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:11.081 INFO:tasks.rados.rados.0.smithi170.stdout:487: done (0 left) 2024-03-27T09:37:11.082 INFO:tasks.rados.rados.0.smithi170.stdout:488: write (excl) oid 9 current snap is 64 2024-03-27T09:37:11.082 INFO:tasks.rados.rados.0.smithi170.stdout:488: seq_num 240 ranges {433680=466703,1577796=455884} 2024-03-27T09:37:11.086 INFO:tasks.rados.rados.0.smithi170.stdout:488: writing smithi17031214-9 from 433680 to 900383 tid 1 2024-03-27T09:37:11.093 INFO:tasks.rados.rados.0.smithi170.stdout:488: writing smithi17031214-9 from 1577796 to 2033680 tid 2 2024-03-27T09:37:11.093 INFO:tasks.rados.rados.0.smithi170.stdout:489: write oid 41 current snap is 64 2024-03-27T09:37:11.093 INFO:tasks.rados.rados.0.smithi170.stdout:489: seq_num 241 ranges {450487=541952,1636278=414209} 2024-03-27T09:37:11.098 INFO:tasks.rados.rados.0.smithi170.stdout:489: writing smithi17031214-41 from 450487 to 992439 tid 1 2024-03-27T09:37:11.105 INFO:tasks.rados.rados.0.smithi170.stdout:489: writing smithi17031214-41 from 1636278 to 2050487 tid 2 2024-03-27T09:37:11.106 INFO:tasks.rados.rados.0.smithi170.stdout:490: snap_create 2024-03-27T09:37:11.192 INFO:tasks.rados.rados.0.smithi170.stdout:488: finishing write tid 1 to smithi17031214-9 2024-03-27T09:37:11.198 INFO:tasks.rados.rados.0.smithi170.stdout:489: finishing write tid 1 to smithi17031214-41 2024-03-27T09:37:11.214 INFO:tasks.rados.rados.0.smithi170.stdout:488: finishing write tid 2 to smithi17031214-9 2024-03-27T09:37:11.218 INFO:tasks.rados.rados.0.smithi170.stdout:489: finishing write tid 2 to smithi17031214-41 2024-03-27T09:37:11.231 INFO:tasks.rados.rados.0.smithi170.stdout:488: finishing write tid 3 to smithi17031214-9 2024-03-27T09:37:11.233 INFO:tasks.rados.rados.0.smithi170.stdout:488: finishing write tid 4 to smithi17031214-9 2024-03-27T09:37:11.233 INFO:tasks.rados.rados.0.smithi170.stdout:488: oid 9 updating version 0 to 138 2024-03-27T09:37:11.234 INFO:tasks.rados.rados.0.smithi170.stdout:488: oid 9 version 138 is already newer than 137 2024-03-27T09:37:11.234 INFO:tasks.rados.rados.0.smithi170.stdout:488: oid 9 version 138 is already newer than 136 2024-03-27T09:37:11.234 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 138 (ObjNum 240 snap 64 seq_num 240) dirty exists 2024-03-27T09:37:11.234 INFO:tasks.rados.rados.0.smithi170.stdout:488: left oid 9 (ObjNum 240 snap 64 seq_num 240) 2024-03-27T09:37:11.236 INFO:tasks.rados.rados.0.smithi170.stdout:489: finishing write tid 3 to smithi17031214-41 2024-03-27T09:37:11.238 INFO:tasks.rados.rados.0.smithi170.stdout:489: finishing write tid 4 to smithi17031214-41 2024-03-27T09:37:11.239 INFO:tasks.rados.rados.0.smithi170.stdout:489: oid 41 updating version 0 to 128 2024-03-27T09:37:11.239 INFO:tasks.rados.rados.0.smithi170.stdout:489: oid 41 version 128 is already newer than 127 2024-03-27T09:37:11.239 INFO:tasks.rados.rados.0.smithi170.stdout:489: oid 41 version 128 is already newer than 126 2024-03-27T09:37:11.239 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 128 (ObjNum 241 snap 64 seq_num 241) dirty exists 2024-03-27T09:37:11.239 INFO:tasks.rados.rados.0.smithi170.stdout:489: left oid 41 (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:12.085 INFO:tasks.rados.rados.0.smithi170.stdout:488: done (2 left) 2024-03-27T09:37:12.086 INFO:tasks.rados.rados.0.smithi170.stdout:489: done (1 left) 2024-03-27T09:37:12.086 INFO:tasks.rados.rados.0.smithi170.stdout:490: done (0 left) 2024-03-27T09:37:12.086 INFO:tasks.rados.rados.0.smithi170.stdout:491: snap_remove snap 59 2024-03-27T09:37:12.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:12.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:12.428+0000 7fbfab39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:12.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:12.434+0000 7fbfab39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:12.459 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:12.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:12.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:12.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s 2024-03-27T09:37:13.053 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:13.056 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:13.086 INFO:tasks.rados.rados.0.smithi170.stdout:491: done (0 left) 2024-03-27T09:37:13.086 INFO:tasks.rados.rados.0.smithi170.stdout:492: snap_create 2024-03-27T09:37:14.090 INFO:tasks.rados.rados.0.smithi170.stdout:492: done (0 left) 2024-03-27T09:37:14.090 INFO:tasks.rados.rados.0.smithi170.stdout:493: read oid 32 snap -1 2024-03-27T09:37:14.090 INFO:tasks.rados.rados.0.smithi170.stdout:493: expect (ObjNum 209 snap 44 seq_num 209) 2024-03-27T09:37:14.090 INFO:tasks.rados.rados.0.smithi170.stdout:494: write oid 40 current snap is 66 2024-03-27T09:37:14.090 INFO:tasks.rados.rados.0.smithi170.stdout:494: seq_num 242 ranges {467294=617201,1578407=488887} 2024-03-27T09:37:14.096 INFO:tasks.rados.rados.0.smithi170.stdout:494: writing smithi17031214-40 from 467294 to 1084495 tid 1 2024-03-27T09:37:14.102 INFO:tasks.rados.rados.0.smithi170.stdout:494: writing smithi17031214-40 from 1578407 to 2067294 tid 2 2024-03-27T09:37:14.103 INFO:tasks.rados.rados.0.smithi170.stdout:495: snap_create 2024-03-27T09:37:14.165 INFO:tasks.rados.rados.0.smithi170.stdout:493: expect (ObjNum 209 snap 44 seq_num 209) 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: finishing write tid 1 to smithi17031214-40 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: finishing write tid 2 to smithi17031214-40 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: finishing write tid 3 to smithi17031214-40 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: finishing write tid 4 to smithi17031214-40 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: oid 40 updating version 0 to 145 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: oid 40 version 145 is already newer than 144 2024-03-27T09:37:14.353 INFO:tasks.rados.rados.0.smithi170.stdout:494: oid 40 updating version 145 to 146 2024-03-27T09:37:14.354 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 146 (ObjNum 242 snap 66 seq_num 242) dirty exists 2024-03-27T09:37:14.354 INFO:tasks.rados.rados.0.smithi170.stdout:494: left oid 40 (ObjNum 242 snap 66 seq_num 242) 2024-03-27T09:37:14.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:14.619 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:14.616+0000 7f2514224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:14.622 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:14.622+0000 7f2514224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:14.646 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:14.692 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:493: done (2 left) 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:494: done (1 left) 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:495: done (0 left) 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:496: read oid 38 snap -1 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:496: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:37:15.095 INFO:tasks.rados.rados.0.smithi170.stdout:497: snap_remove snap 56 2024-03-27T09:37:15.954 INFO:tasks.rados.rados.0.smithi170.stdout:497: done (1 left) 2024-03-27T09:37:15.955 INFO:tasks.rados.rados.0.smithi170.stdout:498: write oid 9 current snap is 67 2024-03-27T09:37:15.955 INFO:tasks.rados.rados.0.smithi170.stdout:498: seq_num 243 ranges {484101=692450,1636889=447212} 2024-03-27T09:37:15.961 INFO:tasks.rados.rados.0.smithi170.stdout:498: writing smithi17031214-9 from 484101 to 1176551 tid 1 2024-03-27T09:37:15.966 INFO:tasks.rados.rados.0.smithi170.stdout:498: writing smithi17031214-9 from 1636889 to 2084101 tid 2 2024-03-27T09:37:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:499: write oid 34 current snap is 67 2024-03-27T09:37:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:499: seq_num 244 ranges {500908=484052,1411724=689184} 2024-03-27T09:37:15.972 INFO:tasks.rados.rados.0.smithi170.stdout:499: writing smithi17031214-34 from 500908 to 984960 tid 1 2024-03-27T09:37:15.979 INFO:tasks.rados.rados.0.smithi170.stdout:499: writing smithi17031214-34 from 1411724 to 2100908 tid 2 2024-03-27T09:37:15.979 INFO:tasks.rados.rados.0.smithi170.stdout:500: snap_create 2024-03-27T09:37:15.980 INFO:tasks.rados.rados.0.smithi170.stdout:496: expect (ObjNum 130 snap 15 seq_num 130) 2024-03-27T09:37:16.091 INFO:tasks.rados.rados.0.smithi170.stdout:498: finishing write tid 1 to smithi17031214-9 2024-03-27T09:37:16.107 INFO:tasks.rados.rados.0.smithi170.stdout:498: finishing write tid 2 to smithi17031214-9 2024-03-27T09:37:16.110 INFO:tasks.rados.rados.0.smithi170.stdout:498: finishing write tid 3 to smithi17031214-9 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:498: finishing write tid 4 to smithi17031214-9 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:498: oid 9 updating version 0 to 154 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:498: oid 9 version 154 is already newer than 154 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:498: oid 9 version 154 is already newer than 152 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 154 (ObjNum 243 snap 67 seq_num 243) dirty exists 2024-03-27T09:37:16.115 INFO:tasks.rados.rados.0.smithi170.stdout:498: left oid 9 (ObjNum 243 snap 67 seq_num 243) 2024-03-27T09:37:16.146 INFO:tasks.rados.rados.0.smithi170.stdout:499: finishing write tid 1 to smithi17031214-34 2024-03-27T09:37:16.166 INFO:tasks.rados.rados.0.smithi170.stdout:499: finishing write tid 2 to smithi17031214-34 2024-03-27T09:37:16.179 INFO:tasks.rados.rados.0.smithi170.stdout:499: finishing write tid 3 to smithi17031214-34 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:499: finishing write tid 4 to smithi17031214-34 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:499: oid 34 updating version 0 to 157 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:499: oid 34 version 157 is already newer than 156 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:499: oid 34 updating version 157 to 158 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 158 (ObjNum 244 snap 67 seq_num 244) dirty exists 2024-03-27T09:37:16.181 INFO:tasks.rados.rados.0.smithi170.stdout:499: left oid 34 (ObjNum 244 snap 67 seq_num 244) 2024-03-27T09:37:16.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:16.807 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:16.805+0000 7f39147d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:16.810+0000 7f39147d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:16.835 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:16.884 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:496: done (3 left) 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:498: done (2 left) 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:499: done (1 left) 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:500: done (0 left) 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:501: write (excl) oid 44 current snap is 68 2024-03-27T09:37:16.955 INFO:tasks.rados.rados.0.smithi170.stdout:501: seq_num 245 ranges {517715=559301,1870206=247509} 2024-03-27T09:37:16.960 INFO:tasks.rados.rados.0.smithi170.stdout:501: writing smithi17031214-44 from 517715 to 1077016 tid 1 2024-03-27T09:37:16.966 INFO:tasks.rados.rados.0.smithi170.stdout:501: writing smithi17031214-44 from 1870206 to 2117715 tid 2 2024-03-27T09:37:16.966 INFO:tasks.rados.rados.0.smithi170.stdout:502: write oid 49 current snap is 68 2024-03-27T09:37:16.966 INFO:tasks.rados.rados.0.smithi170.stdout:502: seq_num 246 ranges {534522=634550,1812335=322187} 2024-03-27T09:37:16.972 INFO:tasks.rados.rados.0.smithi170.stdout:502: writing smithi17031214-49 from 534522 to 1169072 tid 1 2024-03-27T09:37:16.978 INFO:tasks.rados.rados.0.smithi170.stdout:502: writing smithi17031214-49 from 1812335 to 2134522 tid 2 2024-03-27T09:37:16.978 INFO:tasks.rados.rados.0.smithi170.stdout:503: read oid 17 snap -1 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:504: read oid 34 snap -1 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:505: read oid 21 snap -1 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:505: expect deleted 2024-03-27T09:37:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:506: delete oid 30 current snap is 68 2024-03-27T09:37:17.028 INFO:tasks.rados.rados.0.smithi170.stdout:506: done (5 left) 2024-03-27T09:37:17.028 INFO:tasks.rados.rados.0.smithi170.stdout:507: write (excl) oid 31 current snap is 68 2024-03-27T09:37:17.028 INFO:tasks.rados.rados.0.smithi170.stdout:507: seq_num 248 ranges {568136=785048,1929299=238837} 2024-03-27T09:37:17.037 INFO:tasks.rados.rados.0.smithi170.stdout:507: writing smithi17031214-31 from 568136 to 1353184 tid 1 2024-03-27T09:37:17.042 INFO:tasks.rados.rados.0.smithi170.stdout:507: writing smithi17031214-31 from 1929299 to 2168136 tid 2 2024-03-27T09:37:17.043 INFO:tasks.rados.rados.0.smithi170.stdout:508: write (excl) oid 13 current snap is 68 2024-03-27T09:37:17.043 INFO:tasks.rados.rados.0.smithi170.stdout:508: seq_num 249 ranges {584943=460297,1587781=597162} 2024-03-27T09:37:17.048 INFO:tasks.rados.rados.0.smithi170.stdout:508: writing smithi17031214-13 from 584943 to 1045240 tid 1 2024-03-27T09:37:17.057 INFO:tasks.rados.rados.0.smithi170.stdout:508: writing smithi17031214-13 from 1587781 to 2184943 tid 2 2024-03-27T09:37:17.058 INFO:tasks.rados.rados.0.smithi170.stdout:509: snap_create 2024-03-27T09:37:17.058 INFO:tasks.rados.rados.0.smithi170.stdout:504: expect (ObjNum 244 snap 67 seq_num 244) 2024-03-27T09:37:17.168 INFO:tasks.rados.rados.0.smithi170.stdout:501: finishing write tid 1 to smithi17031214-44 2024-03-27T09:37:17.168 INFO:tasks.rados.rados.0.smithi170.stdout:502: finishing write tid 1 to smithi17031214-49 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: finishing write tid 2 to smithi17031214-49 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: finishing write tid 3 to smithi17031214-49 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: finishing write tid 4 to smithi17031214-49 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: oid 49 updating version 0 to 105 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: oid 49 updating version 105 to 106 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: oid 49 version 106 is already newer than 105 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 106 (ObjNum 246 snap 68 seq_num 246) dirty exists 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:502: left oid 49 (ObjNum 246 snap 68 seq_num 246) 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:507: finishing write tid 1 to smithi17031214-31 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:501: finishing write tid 2 to smithi17031214-44 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:508: finishing write tid 1 to smithi17031214-13 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:507: finishing write tid 2 to smithi17031214-31 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:501: finishing write tid 3 to smithi17031214-44 2024-03-27T09:37:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:501: finishing write tid 4 to smithi17031214-44 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:501: oid 44 updating version 0 to 173 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:501: oid 44 version 173 is already newer than 172 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:501: oid 44 version 173 is already newer than 171 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 173 (ObjNum 245 snap 68 seq_num 245) dirty exists 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:501: left oid 44 (ObjNum 245 snap 68 seq_num 245) 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:507: finishing write tid 3 to smithi17031214-31 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:508: finishing write tid 2 to smithi17031214-13 2024-03-27T09:37:17.170 INFO:tasks.rados.rados.0.smithi170.stdout:503: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:17.479 INFO:tasks.rados.rados.0.smithi170.stdout:507: finishing write tid 4 to smithi17031214-31 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:507: oid 31 updating version 0 to 140 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:507: oid 31 updating version 140 to 142 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:507: oid 31 version 142 is already newer than 141 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 142 (ObjNum 248 snap 68 seq_num 248) dirty exists 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:507: left oid 31 (ObjNum 248 snap 68 seq_num 248) 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: finishing write tid 3 to smithi17031214-13 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: finishing write tid 4 to smithi17031214-13 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: oid 13 updating version 0 to 129 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: oid 13 updating version 129 to 131 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: oid 13 version 131 is already newer than 130 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 131 (ObjNum 249 snap 68 seq_num 249) dirty exists 2024-03-27T09:37:17.480 INFO:tasks.rados.rados.0.smithi170.stdout:508: left oid 13 (ObjNum 249 snap 68 seq_num 249) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:501: done (7 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:502: done (6 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:503: done (5 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:504: done (4 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:505: done (3 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:507: done (2 left) 2024-03-27T09:37:17.957 INFO:tasks.rados.rados.0.smithi170.stdout:508: done (1 left) 2024-03-27T09:37:17.958 INFO:tasks.rados.rados.0.smithi170.stdout:509: done (0 left) 2024-03-27T09:37:17.958 INFO:tasks.rados.rados.0.smithi170.stdout:510: write oid 18 current snap is 69 2024-03-27T09:37:17.958 INFO:tasks.rados.rados.0.smithi170.stdout:510: seq_num 250 ranges {601750=535546,1929910=271840} 2024-03-27T09:37:17.964 INFO:tasks.rados.rados.0.smithi170.stdout:510: writing smithi17031214-18 from 601750 to 1137296 tid 1 2024-03-27T09:37:17.970 INFO:tasks.rados.rados.0.smithi170.stdout:510: writing smithi17031214-18 from 1929910 to 2201750 tid 2 2024-03-27T09:37:17.970 INFO:tasks.rados.rados.0.smithi170.stdout:511: read oid 47 snap -1 2024-03-27T09:37:17.970 INFO:tasks.rados.rados.0.smithi170.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2024-03-27T09:37:17.970 INFO:tasks.rados.rados.0.smithi170.stdout:512: snap_remove snap 62 2024-03-27T09:37:18.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:18.157 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:18.160 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:18.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~88s 2024-03-27T09:37:18.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:512: done (2 left) 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:513: read oid 41 snap -1 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:514: delete oid 31 current snap is 69 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:510: finishing write tid 1 to smithi17031214-18 2024-03-27T09:37:18.959 INFO:tasks.rados.rados.0.smithi170.stdout:511: expect (ObjNum 193 snap 38 seq_num 193) 2024-03-27T09:37:19.022 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:19.019+0000 7fc03f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:19.027 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:19.026+0000 7fc03f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:19.064 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:19.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:19.155 INFO:tasks.rados.rados.0.smithi170.stdout:510: finishing write tid 2 to smithi17031214-18 2024-03-27T09:37:19.155 INFO:tasks.rados.rados.0.smithi170.stdout:510: finishing write tid 3 to smithi17031214-18 2024-03-27T09:37:19.155 INFO:tasks.rados.rados.0.smithi170.stdout:510: finishing write tid 4 to smithi17031214-18 2024-03-27T09:37:19.155 INFO:tasks.rados.rados.0.smithi170.stdout:510: oid 18 updating version 0 to 136 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:510: oid 18 version 136 is already newer than 135 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:510: oid 18 version 136 is already newer than 134 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 136 (ObjNum 250 snap 69 seq_num 250) dirty exists 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:510: left oid 18 (ObjNum 250 snap 69 seq_num 250) 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:510: done (3 left) 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:511: done (2 left) 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:514: done (1 left) 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:515: write oid 40 current snap is 69 2024-03-27T09:37:19.156 INFO:tasks.rados.rados.0.smithi170.stdout:515: seq_num 252 ranges {635364=402397,1763227=472137} 2024-03-27T09:37:19.160 INFO:tasks.rados.rados.0.smithi170.stdout:515: writing smithi17031214-40 from 635364 to 1037761 tid 1 2024-03-27T09:37:19.168 INFO:tasks.rados.rados.0.smithi170.stdout:515: writing smithi17031214-40 from 1763227 to 2235364 tid 2 2024-03-27T09:37:19.168 INFO:tasks.rados.rados.0.smithi170.stdout:516: snap_create 2024-03-27T09:37:19.168 INFO:tasks.rados.rados.0.smithi170.stdout:513: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: finishing write tid 1 to smithi17031214-40 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: finishing write tid 2 to smithi17031214-40 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: finishing write tid 3 to smithi17031214-40 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: finishing write tid 4 to smithi17031214-40 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: oid 40 updating version 0 to 167 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: oid 40 updating version 167 to 168 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: oid 40 version 168 is already newer than 166 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 168 (ObjNum 252 snap 69 seq_num 252) dirty exists 2024-03-27T09:37:19.287 INFO:tasks.rados.rados.0.smithi170.stdout:515: left oid 40 (ObjNum 252 snap 69 seq_num 252) 2024-03-27T09:37:19.963 INFO:tasks.rados.rados.0.smithi170.stdout:513: done (2 left) 2024-03-27T09:37:19.963 INFO:tasks.rados.rados.0.smithi170.stdout:515: done (1 left) 2024-03-27T09:37:19.963 INFO:tasks.rados.rados.0.smithi170.stdout:516: done (0 left) 2024-03-27T09:37:19.963 INFO:tasks.rados.rados.0.smithi170.stdout:517: snap_remove snap 60 2024-03-27T09:37:20.964 INFO:tasks.rados.rados.0.smithi170.stdout:517: done (0 left) 2024-03-27T09:37:20.965 INFO:tasks.rados.rados.0.smithi170.stdout:518: snap_remove snap 39 2024-03-27T09:37:21.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:21.232 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:21.230+0000 7f4252b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:21.236 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:21.235+0000 7f4252b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:21.260 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:21.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:21.971 INFO:tasks.rados.rados.0.smithi170.stdout:518: done (0 left) 2024-03-27T09:37:21.971 INFO:tasks.rados.rados.0.smithi170.stdout:519: write (excl) oid 22 current snap is 70 2024-03-27T09:37:21.972 INFO:tasks.rados.rados.0.smithi170.stdout:519: seq_num 253 ranges {652171=477646,1821709=430462} 2024-03-27T09:37:21.977 INFO:tasks.rados.rados.0.smithi170.stdout:519: writing smithi17031214-22 from 652171 to 1129817 tid 1 2024-03-27T09:37:21.984 INFO:tasks.rados.rados.0.smithi170.stdout:519: writing smithi17031214-22 from 1821709 to 2252171 tid 2 2024-03-27T09:37:21.984 INFO:tasks.rados.rados.0.smithi170.stdout:520: read oid 23 snap 50 2024-03-27T09:37:21.984 INFO:tasks.rados.rados.0.smithi170.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2024-03-27T09:37:21.984 INFO:tasks.rados.rados.0.smithi170.stdout:521: read oid 16 snap -1 2024-03-27T09:37:21.985 INFO:tasks.rados.rados.0.smithi170.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:21.985 INFO:tasks.rados.rados.0.smithi170.stdout:522: read oid 1 snap -1 2024-03-27T09:37:21.985 INFO:tasks.rados.rados.0.smithi170.stdout:522: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:37:21.985 INFO:tasks.rados.rados.0.smithi170.stdout:523: delete oid 45 current snap is 70 2024-03-27T09:37:22.066 INFO:tasks.rados.rados.0.smithi170.stdout:520: expect (ObjNum 206 snap 44 seq_num 206) 2024-03-27T09:37:22.205 INFO:tasks.rados.rados.0.smithi170.stdout:519: finishing write tid 1 to smithi17031214-22 2024-03-27T09:37:22.205 INFO:tasks.rados.rados.0.smithi170.stdout:521: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:22.317 INFO:tasks.rados.rados.0.smithi170.stdout:519: finishing write tid 2 to smithi17031214-22 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: finishing write tid 3 to smithi17031214-22 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: finishing write tid 4 to smithi17031214-22 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: oid 22 updating version 0 to 53 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: oid 22 updating version 53 to 55 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: oid 22 version 55 is already newer than 55 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 55 (ObjNum 253 snap 70 seq_num 253) dirty exists 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:519: left oid 22 (ObjNum 253 snap 70 seq_num 253) 2024-03-27T09:37:22.318 INFO:tasks.rados.rados.0.smithi170.stdout:522: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:519: done (4 left) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:520: done (3 left) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:521: done (2 left) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:522: done (1 left) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:523: done (0 left) 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:524: write (excl) oid 18 current snap is 70 2024-03-27T09:37:22.393 INFO:tasks.rados.rados.0.smithi170.stdout:524: seq_num 255 ranges {685785=628144,1822320=463465} 2024-03-27T09:37:22.400 INFO:tasks.rados.rados.0.smithi170.stdout:524: writing smithi17031214-18 from 685785 to 1313929 tid 1 2024-03-27T09:37:22.406 INFO:tasks.rados.rados.0.smithi170.stdout:524: writing smithi17031214-18 from 1822320 to 2285785 tid 2 2024-03-27T09:37:22.406 INFO:tasks.rados.rados.0.smithi170.stdout:525: snap_remove snap 65 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:525: done (1 left) 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:526: read oid 41 snap 66 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:527: read oid 15 snap -1 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:524: finishing write tid 1 to smithi17031214-18 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:528: read oid 7 snap 46 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2024-03-27T09:37:22.975 INFO:tasks.rados.rados.0.smithi170.stdout:529: write oid 26 current snap is 70 2024-03-27T09:37:22.976 INFO:tasks.rados.rados.0.smithi170.stdout:524: finishing write tid 2 to smithi17031214-18 2024-03-27T09:37:22.976 INFO:tasks.rados.rados.0.smithi170.stdout:529: seq_num 256 ranges {702592=703393,1880802=421790} 2024-03-27T09:37:22.982 INFO:tasks.rados.rados.0.smithi170.stdout:529: writing smithi17031214-26 from 702592 to 1405985 tid 1 2024-03-27T09:37:22.988 INFO:tasks.rados.rados.0.smithi170.stdout:529: writing smithi17031214-26 from 1880802 to 2302592 tid 2 2024-03-27T09:37:22.988 INFO:tasks.rados.rados.0.smithi170.stdout:530: snap_remove snap 64 2024-03-27T09:37:22.988 INFO:tasks.rados.rados.0.smithi170.stdout:524: finishing write tid 3 to smithi17031214-18 2024-03-27T09:37:23.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:23.260 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:23.264 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:23.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:23.411+0000 7ff7eddd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:23.417 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:23.416+0000 7ff7eddd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:23.441 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:23.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:530: done (5 left) 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:531: write oid 9 current snap is 70 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:524: finishing write tid 4 to smithi17031214-18 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:524: oid 18 updating version 0 to 153 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:524: oid 18 updating version 153 to 154 2024-03-27T09:37:23.978 INFO:tasks.rados.rados.0.smithi170.stdout:524: oid 18 version 154 is already newer than 152 2024-03-27T09:37:23.979 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 154 (ObjNum 255 snap 70 seq_num 255) dirty exists 2024-03-27T09:37:23.979 INFO:tasks.rados.rados.0.smithi170.stdout:524: left oid 18 (ObjNum 255 snap 70 seq_num 255) 2024-03-27T09:37:23.979 INFO:tasks.rados.rados.0.smithi170.stdout:531: seq_num 257 ranges {719399=778642,1939284=380115} 2024-03-27T09:37:23.986 INFO:tasks.rados.rados.0.smithi170.stdout:531: writing smithi17031214-9 from 719399 to 1498041 tid 1 2024-03-27T09:37:23.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~94s 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:531: writing smithi17031214-9 from 1939284 to 2319399 tid 2 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:524: done (5 left) 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:532: read oid 5 snap 54 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:532: expect (ObjNum 126 snap 15 seq_num 126) 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:533: delete oid 13 current snap is 70 2024-03-27T09:37:23.991 INFO:tasks.rados.rados.0.smithi170.stdout:527: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:24.123 INFO:tasks.rados.rados.0.smithi170.stdout:526: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:24.242 INFO:tasks.rados.rados.0.smithi170.stdout:528: expect (ObjNum 167 snap 26 seq_num 167) 2024-03-27T09:37:24.397 INFO:tasks.rados.rados.0.smithi170.stdout:529: finishing write tid 1 to smithi17031214-26 2024-03-27T09:37:24.397 INFO:tasks.rados.rados.0.smithi170.stdout:529: finishing write tid 2 to smithi17031214-26 2024-03-27T09:37:24.397 INFO:tasks.rados.rados.0.smithi170.stdout:529: finishing write tid 3 to smithi17031214-26 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: finishing write tid 4 to smithi17031214-26 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: oid 26 updating version 0 to 80 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: oid 26 updating version 80 to 82 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: oid 26 version 82 is already newer than 82 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 82 (ObjNum 256 snap 70 seq_num 256) dirty exists 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: left oid 26 (ObjNum 256 snap 70 seq_num 256) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:526: done (6 left) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:527: done (5 left) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:528: done (4 left) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:529: done (3 left) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:533: done (2 left) 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:534: write (excl) oid 26 current snap is 70 2024-03-27T09:37:24.398 INFO:tasks.rados.rados.0.smithi170.stdout:532: expect (ObjNum 126 snap 15 seq_num 126) 2024-03-27T09:37:24.492 INFO:tasks.rados.rados.0.smithi170.stdout:531: finishing write tid 1 to smithi17031214-9 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: finishing write tid 2 to smithi17031214-9 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: finishing write tid 3 to smithi17031214-9 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: finishing write tid 4 to smithi17031214-9 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: oid 9 updating version 0 to 189 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: oid 9 updating version 189 to 190 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: oid 9 version 190 is already newer than 188 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 190 (ObjNum 257 snap 70 seq_num 257) dirty exists 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:531: left oid 9 (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:24.493 INFO:tasks.rados.rados.0.smithi170.stdout:534: seq_num 259 ranges {753013=645493,2056248=296765} 2024-03-27T09:37:24.500 INFO:tasks.rados.rados.0.smithi170.stdout:534: writing smithi17031214-26 from 753013 to 1398506 tid 1 2024-03-27T09:37:24.505 INFO:tasks.rados.rados.0.smithi170.stdout:534: writing smithi17031214-26 from 2056248 to 2353013 tid 2 2024-03-27T09:37:24.505 INFO:tasks.rados.rados.0.smithi170.stdout:531: done (2 left) 2024-03-27T09:37:24.505 INFO:tasks.rados.rados.0.smithi170.stdout:532: done (1 left) 2024-03-27T09:37:24.505 INFO:tasks.rados.rados.0.smithi170.stdout:535: write (excl) oid 6 current snap is 70 2024-03-27T09:37:24.505 INFO:tasks.rados.rados.0.smithi170.stdout:535: seq_num 260 ranges {769820=720742,2114730=255090} 2024-03-27T09:37:24.513 INFO:tasks.rados.rados.0.smithi170.stdout:535: writing smithi17031214-6 from 769820 to 1490562 tid 1 2024-03-27T09:37:24.518 INFO:tasks.rados.rados.0.smithi170.stdout:535: writing smithi17031214-6 from 2114730 to 2369820 tid 2 2024-03-27T09:37:24.518 INFO:tasks.rados.rados.0.smithi170.stdout:536: delete oid 31 current snap is 70 2024-03-27T09:37:24.526 INFO:tasks.rados.rados.0.smithi170.stdout:536: done (2 left) 2024-03-27T09:37:24.526 INFO:tasks.rados.rados.0.smithi170.stdout:537: read oid 13 snap -1 2024-03-27T09:37:24.526 INFO:tasks.rados.rados.0.smithi170.stdout:537: expect deleted 2024-03-27T09:37:24.526 INFO:tasks.rados.rados.0.smithi170.stdout:538: write oid 5 current snap is 70 2024-03-27T09:37:24.526 INFO:tasks.rados.rados.0.smithi170.stdout:538: seq_num 262 ranges {403434=471240,1315341=400756} 2024-03-27T09:37:24.533 INFO:tasks.rados.rados.0.smithi170.stdout:538: writing smithi17031214-5 from 403434 to 874674 tid 1 2024-03-27T09:37:24.539 INFO:tasks.rados.rados.0.smithi170.stdout:538: writing smithi17031214-5 from 1315341 to 1716097 tid 2 2024-03-27T09:37:24.539 INFO:tasks.rados.rados.0.smithi170.stdout:539: snap_remove snap 54 2024-03-27T09:37:24.981 INFO:tasks.rados.rados.0.smithi170.stdout:539: done (4 left) 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:540: delete oid 16 current snap is 70 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: finishing write tid 1 to smithi17031214-26 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: finishing write tid 2 to smithi17031214-26 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:535: finishing write tid 1 to smithi17031214-6 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:538: finishing write tid 1 to smithi17031214-5 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: finishing write tid 3 to smithi17031214-26 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: finishing write tid 4 to smithi17031214-26 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: oid 26 updating version 0 to 86 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: oid 26 updating version 86 to 87 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: oid 26 version 87 is already newer than 85 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 87 (ObjNum 259 snap 70 seq_num 259) dirty exists 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:534: left oid 26 (ObjNum 259 snap 70 seq_num 259) 2024-03-27T09:37:24.982 INFO:tasks.rados.rados.0.smithi170.stdout:535: finishing write tid 2 to smithi17031214-6 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: finishing write tid 2 to smithi17031214-5 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: finishing write tid 3 to smithi17031214-6 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: finishing write tid 4 to smithi17031214-6 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: oid 6 updating version 0 to 60 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: oid 6 updating version 60 to 61 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: oid 6 updating version 61 to 62 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 62 (ObjNum 260 snap 70 seq_num 260) dirty exists 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:535: left oid 6 (ObjNum 260 snap 70 seq_num 260) 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: finishing write tid 3 to smithi17031214-5 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: finishing write tid 4 to smithi17031214-5 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: oid 5 updating version 0 to 64 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: oid 5 updating version 64 to 65 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:538: oid 5 updating version 65 to 66 2024-03-27T09:37:24.983 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 66 (ObjNum 262 snap 70 seq_num 262) dirty exists 2024-03-27T09:37:24.984 INFO:tasks.rados.rados.0.smithi170.stdout:538: left oid 5 (ObjNum 262 snap 70 seq_num 262) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:534: done (4 left) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:535: done (3 left) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:537: done (2 left) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:538: done (1 left) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:540: done (0 left) 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:541: write (excl) oid 44 current snap is 70 2024-03-27T09:37:25.073 INFO:tasks.rados.rados.0.smithi170.stdout:541: seq_num 264 ranges {437048=621738,1832305=404425} 2024-03-27T09:37:25.078 INFO:tasks.rados.rados.0.smithi170.stdout:541: writing smithi17031214-44 from 437048 to 1058786 tid 1 2024-03-27T09:37:25.085 INFO:tasks.rados.rados.0.smithi170.stdout:541: writing smithi17031214-44 from 1832305 to 2236730 tid 2 2024-03-27T09:37:25.085 INFO:tasks.rados.rados.0.smithi170.stdout:542: write oid 31 current snap is 70 2024-03-27T09:37:25.085 INFO:tasks.rados.rados.0.smithi170.stdout:542: seq_num 265 ranges {453855=696987,1890787=548083} 2024-03-27T09:37:25.091 INFO:tasks.rados.rados.0.smithi170.stdout:542: writing smithi17031214-31 from 453855 to 1150842 tid 1 2024-03-27T09:37:25.098 INFO:tasks.rados.rados.0.smithi170.stdout:542: writing smithi17031214-31 from 1890787 to 2438870 tid 2 2024-03-27T09:37:25.098 INFO:tasks.rados.rados.0.smithi170.stdout:543: write oid 21 current snap is 70 2024-03-27T09:37:25.099 INFO:tasks.rados.rados.0.smithi170.stdout:543: seq_num 266 ranges {470662=772236,1832916=408094} 2024-03-27T09:37:25.105 INFO:tasks.rados.rados.0.smithi170.stdout:543: writing smithi17031214-21 from 470662 to 1242898 tid 1 2024-03-27T09:37:25.111 INFO:tasks.rados.rados.0.smithi170.stdout:543: writing smithi17031214-21 from 1832916 to 2241010 tid 2 2024-03-27T09:37:25.111 INFO:tasks.rados.rados.0.smithi170.stdout:544: delete oid 45 current snap is 70 2024-03-27T09:37:25.119 INFO:tasks.rados.rados.0.smithi170.stdout:541: finishing write tid 1 to smithi17031214-44 2024-03-27T09:37:25.121 INFO:tasks.rados.rados.0.smithi170.stdout:542: finishing write tid 1 to smithi17031214-31 2024-03-27T09:37:25.147 INFO:tasks.rados.rados.0.smithi170.stdout:542: finishing write tid 2 to smithi17031214-31 2024-03-27T09:37:25.153 INFO:tasks.rados.rados.0.smithi170.stdout:541: finishing write tid 2 to smithi17031214-44 2024-03-27T09:37:25.161 INFO:tasks.rados.rados.0.smithi170.stdout:541: finishing write tid 3 to smithi17031214-44 2024-03-27T09:37:25.164 INFO:tasks.rados.rados.0.smithi170.stdout:541: finishing write tid 4 to smithi17031214-44 2024-03-27T09:37:25.164 INFO:tasks.rados.rados.0.smithi170.stdout:541: oid 44 updating version 0 to 197 2024-03-27T09:37:25.164 INFO:tasks.rados.rados.0.smithi170.stdout:541: oid 44 version 197 is already newer than 197 2024-03-27T09:37:25.165 INFO:tasks.rados.rados.0.smithi170.stdout:541: oid 44 version 197 is already newer than 195 2024-03-27T09:37:25.165 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 197 (ObjNum 264 snap 70 seq_num 264) dirty exists 2024-03-27T09:37:25.165 INFO:tasks.rados.rados.0.smithi170.stdout:541: left oid 44 (ObjNum 264 snap 70 seq_num 264) 2024-03-27T09:37:25.169 INFO:tasks.rados.rados.0.smithi170.stdout:541: done (3 left) 2024-03-27T09:37:25.169 INFO:tasks.rados.rados.0.smithi170.stdout:544: done (2 left) 2024-03-27T09:37:25.169 INFO:tasks.rados.rados.0.smithi170.stdout:545: write oid 5 current snap is 70 2024-03-27T09:37:25.169 INFO:tasks.rados.rados.0.smithi170.stdout:542: finishing write tid 3 to smithi17031214-31 2024-03-27T09:37:25.169 INFO:tasks.rados.rados.0.smithi170.stdout:545: seq_num 268 ranges {504276=639087,1666233=695410} 2024-03-27T09:37:25.175 INFO:tasks.rados.rados.0.smithi170.stdout:545: writing smithi17031214-5 from 504276 to 1143363 tid 1 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:545: writing smithi17031214-5 from 1666233 to 2361643 tid 2 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:546: read oid 43 snap 63 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:546: expect (ObjNum 223 snap 50 seq_num 223) 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:547: write oid 10 current snap is 70 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:542: finishing write tid 4 to smithi17031214-31 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:542: oid 31 updating version 0 to 163 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:542: oid 31 updating version 163 to 165 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:542: oid 31 version 165 is already newer than 164 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 165 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:542: left oid 31 (ObjNum 265 snap 70 seq_num 265) 2024-03-27T09:37:25.183 INFO:tasks.rados.rados.0.smithi170.stdout:547: seq_num 269 ranges {521083=714336,1724715=555421} 2024-03-27T09:37:25.188 INFO:tasks.rados.rados.0.smithi170.stdout:547: writing smithi17031214-10 from 521083 to 1235419 tid 1 2024-03-27T09:37:25.194 INFO:tasks.rados.rados.0.smithi170.stdout:547: writing smithi17031214-10 from 1724715 to 2280136 tid 2 2024-03-27T09:37:25.194 INFO:tasks.rados.rados.0.smithi170.stdout:542: done (4 left) 2024-03-27T09:37:25.194 INFO:tasks.rados.rados.0.smithi170.stdout:548: snap_remove snap 66 2024-03-27T09:37:25.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:25.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:25.594+0000 7f7746a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:25.600 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:25.599+0000 7f7746a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:25.624 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:25.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:25.956 INFO:tasks.rados.rados.0.smithi170.stdout:548: done (4 left) 2024-03-27T09:37:25.956 INFO:tasks.rados.rados.0.smithi170.stdout:549: write oid 1 current snap is 70 2024-03-27T09:37:25.956 INFO:tasks.rados.rados.0.smithi170.stdout:549: seq_num 270 ranges {537890=789585,1783197=699079} 2024-03-27T09:37:25.964 INFO:tasks.rados.rados.0.smithi170.stdout:549: writing smithi17031214-1 from 537890 to 1327475 tid 1 2024-03-27T09:37:25.972 INFO:tasks.rados.rados.0.smithi170.stdout:549: writing smithi17031214-1 from 1783197 to 2482276 tid 2 2024-03-27T09:37:25.972 INFO:tasks.rados.rados.0.smithi170.stdout:550: write (excl) oid 38 current snap is 70 2024-03-27T09:37:25.972 INFO:tasks.rados.rados.0.smithi170.stdout:543: finishing write tid 1 to smithi17031214-21 2024-03-27T09:37:25.972 INFO:tasks.rados.rados.0.smithi170.stdout:550: seq_num 271 ranges {554697=464834,1725326=559090} 2024-03-27T09:37:25.980 INFO:tasks.rados.rados.0.smithi170.stdout:550: writing smithi17031214-38 from 554697 to 1019531 tid 1 2024-03-27T09:37:25.989 INFO:tasks.rados.rados.0.smithi170.stdout:550: writing smithi17031214-38 from 1725326 to 2284416 tid 2 2024-03-27T09:37:25.990 INFO:tasks.rados.rados.0.smithi170.stdout:551: snap_remove snap 46 2024-03-27T09:37:26.988 INFO:tasks.rados.rados.0.smithi170.stdout:551: done (6 left) 2024-03-27T09:37:26.988 INFO:tasks.rados.rados.0.smithi170.stdout:552: write (excl) oid 24 current snap is 70 2024-03-27T09:37:26.988 INFO:tasks.rados.rados.0.smithi170.stdout:543: finishing write tid 2 to smithi17031214-21 2024-03-27T09:37:26.988 INFO:tasks.rados.rados.0.smithi170.stdout:552: seq_num 272 ranges {571504=540083,1783808=702748} 2024-03-27T09:37:26.994 INFO:tasks.rados.rados.0.smithi170.stdout:552: writing smithi17031214-24 from 571504 to 1111587 tid 1 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:552: writing smithi17031214-24 from 1783808 to 2486556 tid 2 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:553: read oid 31 snap 58 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:553: expect (ObjNum 213 snap 45 seq_num 213) 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:554: read oid 44 snap -1 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:554: expect (ObjNum 264 snap 70 seq_num 264) 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:555: delete oid 30 current snap is 70 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:543: finishing write tid 3 to smithi17031214-21 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:543: finishing write tid 4 to smithi17031214-21 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:543: oid 21 updating version 0 to 91 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:543: oid 21 updating version 91 to 92 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:543: oid 21 version 92 is already newer than 90 2024-03-27T09:37:27.003 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 92 (ObjNum 266 snap 70 seq_num 266) dirty exists 2024-03-27T09:37:27.004 INFO:tasks.rados.rados.0.smithi170.stdout:543: left oid 21 (ObjNum 266 snap 70 seq_num 266) 2024-03-27T09:37:27.004 INFO:tasks.rados.rados.0.smithi170.stdout:546: expect (ObjNum 223 snap 50 seq_num 223) 2024-03-27T09:37:27.173 INFO:tasks.rados.rados.0.smithi170.stdout:547: finishing write tid 1 to smithi17031214-10 2024-03-27T09:37:27.173 INFO:tasks.rados.rados.0.smithi170.stdout:545: finishing write tid 1 to smithi17031214-5 2024-03-27T09:37:27.173 INFO:tasks.rados.rados.0.smithi170.stdout:547: finishing write tid 2 to smithi17031214-10 2024-03-27T09:37:27.174 INFO:tasks.rados.rados.0.smithi170.stdout:545: finishing write tid 2 to smithi17031214-5 2024-03-27T09:37:27.174 INFO:tasks.rados.rados.0.smithi170.stdout:543: done (9 left) 2024-03-27T09:37:27.174 INFO:tasks.rados.rados.0.smithi170.stdout:546: done (8 left) 2024-03-27T09:37:27.174 INFO:tasks.rados.rados.0.smithi170.stdout:555: done (7 left) 2024-03-27T09:37:27.174 INFO:tasks.rados.rados.0.smithi170.stdout:556: snap_remove snap 57 2024-03-27T09:37:27.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:27.780 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:27.778+0000 7f296f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:27.784 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:27.783+0000 7f296f8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:27.807 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:27.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:27.991 INFO:tasks.rados.rados.0.smithi170.stdout:556: done (7 left) 2024-03-27T09:37:27.991 INFO:tasks.rados.rados.0.smithi170.stdout:557: read oid 13 snap -1 2024-03-27T09:37:27.991 INFO:tasks.rados.rados.0.smithi170.stdout:557: expect deleted 2024-03-27T09:37:27.991 INFO:tasks.rados.rados.0.smithi170.stdout:558: snap_remove snap 68 2024-03-27T09:37:28.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:28.364 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:28.367 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:558: done (8 left) 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:559: read oid 50 snap -1 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:559: expect (ObjNum 227 snap 55 seq_num 227) 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:560: write oid 40 current snap is 70 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:547: finishing write tid 3 to smithi17031214-10 2024-03-27T09:37:28.993 INFO:tasks.rados.rados.0.smithi170.stdout:547: finishing write tid 4 to smithi17031214-10 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:547: oid 10 updating version 0 to 10 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:547: oid 10 updating version 10 to 12 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:547: oid 10 version 12 is already newer than 11 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 12 (ObjNum 269 snap 70 seq_num 269) dirty exists 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:547: left oid 10 (ObjNum 269 snap 70 seq_num 269) 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: finishing write tid 3 to smithi17031214-5 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: finishing write tid 4 to smithi17031214-5 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: oid 5 updating version 0 to 73 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: oid 5 version 73 is already newer than 73 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: oid 5 version 73 is already newer than 71 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 73 (ObjNum 268 snap 70 seq_num 268) dirty exists 2024-03-27T09:37:28.994 INFO:tasks.rados.rados.0.smithi170.stdout:545: left oid 5 (ObjNum 268 snap 70 seq_num 268) 2024-03-27T09:37:28.995 INFO:tasks.rados.rados.0.smithi170.stdout:560: seq_num 274 ranges {605118=406934,1617125=706417} 2024-03-27T09:37:28.998 INFO:tasks.rados.rados.0.smithi170.stdout:560: writing smithi17031214-40 from 605118 to 1012052 tid 1 2024-03-27T09:37:29.006 INFO:tasks.rados.rados.0.smithi170.stdout:560: writing smithi17031214-40 from 1617125 to 2323542 tid 2 2024-03-27T09:37:29.007 INFO:tasks.rados.rados.0.smithi170.stdout:545: done (9 left) 2024-03-27T09:37:29.007 INFO:tasks.rados.rados.0.smithi170.stdout:547: done (8 left) 2024-03-27T09:37:29.007 INFO:tasks.rados.rados.0.smithi170.stdout:561: snap_remove snap 53 2024-03-27T09:37:29.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~99s 2024-03-27T09:37:29.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:29.967 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:29.964+0000 7f3a88a60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:29.970 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:29.970+0000 7f3a88a60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:29.995 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:29.998 INFO:tasks.rados.rados.0.smithi170.stdout:561: done (8 left) 2024-03-27T09:37:29.998 INFO:tasks.rados.rados.0.smithi170.stdout:562: snap_create 2024-03-27T09:37:29.998 INFO:tasks.rados.rados.0.smithi170.stdout:550: finishing write tid 1 to smithi17031214-38 2024-03-27T09:37:29.998 INFO:tasks.rados.rados.0.smithi170.stdout:549: finishing write tid 1 to smithi17031214-1 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: finishing write tid 2 to smithi17031214-38 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: finishing write tid 2 to smithi17031214-1 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: finishing write tid 3 to smithi17031214-38 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: finishing write tid 4 to smithi17031214-38 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: oid 38 updating version 0 to 96 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: oid 38 updating version 96 to 98 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: oid 38 version 98 is already newer than 97 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 98 (ObjNum 271 snap 70 seq_num 271) dirty exists 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:550: left oid 38 (ObjNum 271 snap 70 seq_num 271) 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: finishing write tid 3 to smithi17031214-1 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: finishing write tid 4 to smithi17031214-1 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: oid 1 updating version 0 to 68 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: oid 1 updating version 68 to 69 2024-03-27T09:37:29.999 INFO:tasks.rados.rados.0.smithi170.stdout:549: oid 1 updating version 69 to 70 2024-03-27T09:37:30.000 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 70 (ObjNum 270 snap 70 seq_num 270) dirty exists 2024-03-27T09:37:30.000 INFO:tasks.rados.rados.0.smithi170.stdout:549: left oid 1 (ObjNum 270 snap 70 seq_num 270) 2024-03-27T09:37:30.000 INFO:tasks.rados.rados.0.smithi170.stdout:554: expect (ObjNum 264 snap 70 seq_num 264) 2024-03-27T09:37:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:30.103 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:30.203 INFO:tasks.rados.rados.0.smithi170.stdout:552: finishing write tid 1 to smithi17031214-24 2024-03-27T09:37:30.203 INFO:tasks.rados.rados.0.smithi170.stdout:553: expect (ObjNum 213 snap 45 seq_num 213) 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: finishing write tid 2 to smithi17031214-24 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: finishing write tid 3 to smithi17031214-24 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: finishing write tid 4 to smithi17031214-24 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: oid 24 updating version 0 to 76 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: oid 24 version 76 is already newer than 76 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: oid 24 version 76 is already newer than 74 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 76 (ObjNum 272 snap 70 seq_num 272) dirty exists 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:552: left oid 24 (ObjNum 272 snap 70 seq_num 272) 2024-03-27T09:37:30.339 INFO:tasks.rados.rados.0.smithi170.stdout:559: expect (ObjNum 227 snap 55 seq_num 227) 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: finishing write tid 1 to smithi17031214-40 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: finishing write tid 2 to smithi17031214-40 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: finishing write tid 3 to smithi17031214-40 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: finishing write tid 4 to smithi17031214-40 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: oid 40 updating version 0 to 228 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: oid 40 version 228 is already newer than 226 2024-03-27T09:37:30.579 INFO:tasks.rados.rados.0.smithi170.stdout:560: oid 40 version 228 is already newer than 228 2024-03-27T09:37:30.580 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 228 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-03-27T09:37:30.580 INFO:tasks.rados.rados.0.smithi170.stdout:560: left oid 40 (ObjNum 274 snap 70 seq_num 274) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:549: done (8 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:550: done (7 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:552: done (6 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:553: done (5 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:554: done (4 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:557: done (3 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:559: done (2 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:560: done (1 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:562: done (0 left) 2024-03-27T09:37:31.001 INFO:tasks.rados.rados.0.smithi170.stdout:563: write oid 6 current snap is 71 2024-03-27T09:37:31.002 INFO:tasks.rados.rados.0.smithi170.stdout:563: seq_num 275 ranges {621925=482183,1559254=566428} 2024-03-27T09:37:31.006 INFO:tasks.rados.rados.0.smithi170.stdout:563: writing smithi17031214-6 from 621925 to 1104108 tid 1 2024-03-27T09:37:31.012 INFO:tasks.rados.rados.0.smithi170.stdout:563: writing smithi17031214-6 from 1559254 to 2125682 tid 2 2024-03-27T09:37:31.012 INFO:tasks.rados.rados.0.smithi170.stdout:564: snap_create 2024-03-27T09:37:31.072 INFO:tasks.rados.rados.0.smithi170.stdout:563: finishing write tid 1 to smithi17031214-6 2024-03-27T09:37:31.090 INFO:tasks.rados.rados.0.smithi170.stdout:563: finishing write tid 2 to smithi17031214-6 2024-03-27T09:37:31.100 INFO:tasks.rados.rados.0.smithi170.stdout:563: finishing write tid 3 to smithi17031214-6 2024-03-27T09:37:31.101 INFO:tasks.rados.rados.0.smithi170.stdout:563: finishing write tid 4 to smithi17031214-6 2024-03-27T09:37:31.101 INFO:tasks.rados.rados.0.smithi170.stdout:563: oid 6 updating version 0 to 89 2024-03-27T09:37:31.101 INFO:tasks.rados.rados.0.smithi170.stdout:563: oid 6 version 89 is already newer than 88 2024-03-27T09:37:31.102 INFO:tasks.rados.rados.0.smithi170.stdout:563: oid 6 updating version 89 to 90 2024-03-27T09:37:31.102 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 90 (ObjNum 275 snap 71 seq_num 275) dirty exists 2024-03-27T09:37:31.102 INFO:tasks.rados.rados.0.smithi170.stdout:563: left oid 6 (ObjNum 275 snap 71 seq_num 275) 2024-03-27T09:37:32.004 INFO:tasks.rados.rados.0.smithi170.stdout:563: done (1 left) 2024-03-27T09:37:32.005 INFO:tasks.rados.rados.0.smithi170.stdout:564: done (0 left) 2024-03-27T09:37:32.005 INFO:tasks.rados.rados.0.smithi170.stdout:565: write oid 22 current snap is 72 2024-03-27T09:37:32.005 INFO:tasks.rados.rados.0.smithi170.stdout:565: seq_num 276 ranges {638732=557432,1617736=710086} 2024-03-27T09:37:32.011 INFO:tasks.rados.rados.0.smithi170.stdout:565: writing smithi17031214-22 from 638732 to 1196164 tid 1 2024-03-27T09:37:32.017 INFO:tasks.rados.rados.0.smithi170.stdout:565: writing smithi17031214-22 from 1617736 to 2327822 tid 2 2024-03-27T09:37:32.017 INFO:tasks.rados.rados.0.smithi170.stdout:566: snap_remove snap 42 2024-03-27T09:37:32.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:32.216 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:32.214+0000 7ff57c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:32.220 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:32.219+0000 7ff57c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:32.244 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:32.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:33.007 INFO:tasks.rados.rados.0.smithi170.stdout:566: done (1 left) 2024-03-27T09:37:33.007 INFO:tasks.rados.rados.0.smithi170.stdout:567: snap_create 2024-03-27T09:37:33.007 INFO:tasks.rados.rados.0.smithi170.stdout:565: finishing write tid 1 to smithi17031214-22 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: finishing write tid 2 to smithi17031214-22 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: finishing write tid 3 to smithi17031214-22 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: finishing write tid 4 to smithi17031214-22 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: oid 22 updating version 0 to 94 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: oid 22 version 94 is already newer than 93 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: oid 22 version 94 is already newer than 92 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 94 (ObjNum 276 snap 72 seq_num 276) dirty exists 2024-03-27T09:37:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:565: left oid 22 (ObjNum 276 snap 72 seq_num 276) 2024-03-27T09:37:33.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:33.467 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:33.470 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:34.012 INFO:tasks.rados.rados.0.smithi170.stdout:565: done (1 left) 2024-03-27T09:37:34.012 INFO:tasks.rados.rados.0.smithi170.stdout:567: done (0 left) 2024-03-27T09:37:34.012 INFO:tasks.rados.rados.0.smithi170.stdout:568: snap_create 2024-03-27T09:37:34.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:34.398 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:34.396+0000 7ff73f406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:34.401 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:34.401+0000 7ff73f406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:34.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:34.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2024-03-27T09:37:35.013 INFO:tasks.rados.rados.0.smithi170.stdout:568: done (0 left) 2024-03-27T09:37:35.013 INFO:tasks.rados.rados.0.smithi170.stdout:569: snap_remove snap 69 2024-03-27T09:37:35.958 INFO:tasks.rados.rados.0.smithi170.stdout:569: done (0 left) 2024-03-27T09:37:35.958 INFO:tasks.rados.rados.0.smithi170.stdout:570: write oid 46 current snap is 74 2024-03-27T09:37:35.959 INFO:tasks.rados.rados.0.smithi170.stdout:570: seq_num 277 ranges {655539=632681,2076218=570097} 2024-03-27T09:37:35.965 INFO:tasks.rados.rados.0.smithi170.stdout:570: writing smithi17031214-46 from 655539 to 1288220 tid 1 2024-03-27T09:37:35.973 INFO:tasks.rados.rados.0.smithi170.stdout:570: writing smithi17031214-46 from 2076218 to 2646315 tid 2 2024-03-27T09:37:35.973 INFO:tasks.rados.rados.0.smithi170.stdout:571: snap_remove snap 50 2024-03-27T09:37:36.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:36.582 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:36.579+0000 7fa99f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:36.585 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:36.584+0000 7fa99f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:36.609 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:36.654 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:36.959 INFO:tasks.rados.rados.0.smithi170.stdout:571: done (1 left) 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:572: delete oid 37 current snap is 74 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: finishing write tid 1 to smithi17031214-46 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: finishing write tid 2 to smithi17031214-46 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: finishing write tid 3 to smithi17031214-46 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: finishing write tid 4 to smithi17031214-46 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: oid 46 updating version 0 to 245 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: oid 46 updating version 245 to 247 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: oid 46 version 247 is already newer than 247 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 247 (ObjNum 277 snap 74 seq_num 277) dirty exists 2024-03-27T09:37:36.960 INFO:tasks.rados.rados.0.smithi170.stdout:570: left oid 46 (ObjNum 277 snap 74 seq_num 277) 2024-03-27T09:37:37.025 INFO:tasks.rados.rados.0.smithi170.stdout:570: done (1 left) 2024-03-27T09:37:37.025 INFO:tasks.rados.rados.0.smithi170.stdout:572: done (0 left) 2024-03-27T09:37:37.025 INFO:tasks.rados.rados.0.smithi170.stdout:573: delete oid 48 current snap is 74 2024-03-27T09:37:37.053 INFO:tasks.rados.rados.0.smithi170.stdout:573: done (0 left) 2024-03-27T09:37:37.053 INFO:tasks.rados.rados.0.smithi170.stdout:574: read oid 9 snap -1 2024-03-27T09:37:37.053 INFO:tasks.rados.rados.0.smithi170.stdout:574: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:37.053 INFO:tasks.rados.rados.0.smithi170.stdout:575: snap_create 2024-03-27T09:37:37.072 INFO:tasks.rados.rados.0.smithi170.stdout:574: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:38.023 INFO:tasks.rados.rados.0.smithi170.stdout:574: done (1 left) 2024-03-27T09:37:38.023 INFO:tasks.rados.rados.0.smithi170.stdout:575: done (0 left) 2024-03-27T09:37:38.023 INFO:tasks.rados.rados.0.smithi170.stdout:576: write oid 45 current snap is 75 2024-03-27T09:37:38.023 INFO:tasks.rados.rados.0.smithi170.stdout:576: seq_num 280 ranges {705960=458428,1735311=717424} 2024-03-27T09:37:38.028 INFO:tasks.rados.rados.0.smithi170.stdout:576: writing smithi17031214-45 from 705960 to 1164388 tid 1 2024-03-27T09:37:38.038 INFO:tasks.rados.rados.0.smithi170.stdout:576: writing smithi17031214-45 from 1735311 to 2452735 tid 2 2024-03-27T09:37:38.038 INFO:tasks.rados.rados.0.smithi170.stdout:577: read oid 2 snap 58 2024-03-27T09:37:38.038 INFO:tasks.rados.rados.0.smithi170.stdout:577: expect deleted 2024-03-27T09:37:38.038 INFO:tasks.rados.rados.0.smithi170.stdout:578: snap_remove snap 70 2024-03-27T09:37:38.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:38.580 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:38.583 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:38.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:38.764 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:38.761+0000 7f1db659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:38.768 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:38.766+0000 7f1db659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:38.792 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:38.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:39.028 INFO:tasks.rados.rados.0.smithi170.stdout:578: done (2 left) 2024-03-27T09:37:39.029 INFO:tasks.rados.rados.0.smithi170.stdout:579: snap_remove snap 74 2024-03-27T09:37:40.030 INFO:tasks.rados.rados.0.smithi170.stdout:579: done (2 left) 2024-03-27T09:37:40.031 INFO:tasks.rados.rados.0.smithi170.stdout:580: write (excl) oid 40 current snap is 75 2024-03-27T09:37:40.031 INFO:tasks.rados.rados.0.smithi170.stdout:576: finishing write tid 1 to smithi17031214-45 2024-03-27T09:37:40.031 INFO:tasks.rados.rados.0.smithi170.stdout:580: seq_num 281 ranges {722767=533677,1793793=461082} 2024-03-27T09:37:40.037 INFO:tasks.rados.rados.0.smithi170.stdout:580: writing smithi17031214-40 from 722767 to 1256444 tid 1 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:580: writing smithi17031214-40 from 1793793 to 2254875 tid 2 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:581: read oid 50 snap -1 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:581: expect (ObjNum 227 snap 55 seq_num 227) 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:582: write oid 19 current snap is 75 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:576: finishing write tid 2 to smithi17031214-45 2024-03-27T09:37:40.043 INFO:tasks.rados.rados.0.smithi170.stdout:576: finishing write tid 3 to smithi17031214-45 2024-03-27T09:37:40.044 INFO:tasks.rados.rados.0.smithi170.stdout:582: seq_num 282 ranges {739574=725279,1968628=721093} 2024-03-27T09:37:40.051 INFO:tasks.rados.rados.0.smithi170.stdout:582: writing smithi17031214-19 from 739574 to 1464853 tid 1 2024-03-27T09:37:40.059 INFO:tasks.rados.rados.0.smithi170.stdout:582: writing smithi17031214-19 from 1968628 to 2689721 tid 2 2024-03-27T09:37:40.059 INFO:tasks.rados.rados.0.smithi170.stdout:583: snap_create 2024-03-27T09:37:40.059 INFO:tasks.rados.rados.0.smithi170.stdout:576: finishing write tid 4 to smithi17031214-45 2024-03-27T09:37:40.060 INFO:tasks.rados.rados.0.smithi170.stdout:576: oid 45 updating version 0 to 204 2024-03-27T09:37:40.060 INFO:tasks.rados.rados.0.smithi170.stdout:576: oid 45 version 204 is already newer than 203 2024-03-27T09:37:40.060 INFO:tasks.rados.rados.0.smithi170.stdout:576: oid 45 version 204 is already newer than 202 2024-03-27T09:37:40.060 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 204 (ObjNum 280 snap 75 seq_num 280) dirty exists 2024-03-27T09:37:40.060 INFO:tasks.rados.rados.0.smithi170.stdout:576: left oid 45 (ObjNum 280 snap 75 seq_num 280) 2024-03-27T09:37:40.137 INFO:tasks.rados.rados.0.smithi170.stdout:582: finishing write tid 1 to smithi17031214-19 2024-03-27T09:37:40.138 INFO:tasks.rados.rados.0.smithi170.stdout:580: finishing write tid 1 to smithi17031214-40 2024-03-27T09:37:40.145 INFO:tasks.rados.rados.0.smithi170.stdout:581: expect (ObjNum 227 snap 55 seq_num 227) 2024-03-27T09:37:40.393 INFO:tasks.rados.rados.0.smithi170.stdout:582: finishing write tid 2 to smithi17031214-19 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:580: finishing write tid 2 to smithi17031214-40 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: finishing write tid 3 to smithi17031214-19 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: finishing write tid 4 to smithi17031214-19 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: oid 19 updating version 0 to 124 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: oid 19 updating version 124 to 126 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: oid 19 version 126 is already newer than 125 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 126 (ObjNum 282 snap 75 seq_num 282) dirty exists 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:582: left oid 19 (ObjNum 282 snap 75 seq_num 282) 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:580: finishing write tid 3 to smithi17031214-40 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:580: finishing write tid 4 to smithi17031214-40 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:580: oid 40 updating version 0 to 256 2024-03-27T09:37:40.394 INFO:tasks.rados.rados.0.smithi170.stdout:580: oid 40 updating version 256 to 257 2024-03-27T09:37:40.395 INFO:tasks.rados.rados.0.smithi170.stdout:580: oid 40 version 257 is already newer than 255 2024-03-27T09:37:40.395 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 257 (ObjNum 281 snap 75 seq_num 281) dirty exists 2024-03-27T09:37:40.395 INFO:tasks.rados.rados.0.smithi170.stdout:580: left oid 40 (ObjNum 281 snap 75 seq_num 281) 2024-03-27T09:37:40.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2024-03-27T09:37:40.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:40.951 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:40.949+0000 7fa1532ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:40.955 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:40.954+0000 7fa1532ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:40.982 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:41.030 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:41.033 INFO:tasks.rados.rados.0.smithi170.stdout:576: done (5 left) 2024-03-27T09:37:41.033 INFO:tasks.rados.rados.0.smithi170.stdout:577: done (4 left) 2024-03-27T09:37:41.033 INFO:tasks.rados.rados.0.smithi170.stdout:580: done (3 left) 2024-03-27T09:37:41.033 INFO:tasks.rados.rados.0.smithi170.stdout:581: done (2 left) 2024-03-27T09:37:41.034 INFO:tasks.rados.rados.0.smithi170.stdout:582: done (1 left) 2024-03-27T09:37:41.034 INFO:tasks.rados.rados.0.smithi170.stdout:583: done (0 left) 2024-03-27T09:37:41.034 INFO:tasks.rados.rados.0.smithi170.stdout:584: write (excl) oid 28 current snap is 76 2024-03-27T09:37:41.034 INFO:tasks.rados.rados.0.smithi170.stdout:584: seq_num 283 ranges {756381=400528,1910757=464751} 2024-03-27T09:37:41.039 INFO:tasks.rados.rados.0.smithi170.stdout:584: writing smithi17031214-28 from 756381 to 1156909 tid 1 2024-03-27T09:37:41.047 INFO:tasks.rados.rados.0.smithi170.stdout:584: writing smithi17031214-28 from 1910757 to 2375508 tid 2 2024-03-27T09:37:41.047 INFO:tasks.rados.rados.0.smithi170.stdout:585: snap_create 2024-03-27T09:37:41.110 INFO:tasks.rados.rados.0.smithi170.stdout:584: finishing write tid 1 to smithi17031214-28 2024-03-27T09:37:41.123 INFO:tasks.rados.rados.0.smithi170.stdout:584: finishing write tid 2 to smithi17031214-28 2024-03-27T09:37:41.127 INFO:tasks.rados.rados.0.smithi170.stdout:584: finishing write tid 3 to smithi17031214-28 2024-03-27T09:37:41.128 INFO:tasks.rados.rados.0.smithi170.stdout:584: finishing write tid 4 to smithi17031214-28 2024-03-27T09:37:41.129 INFO:tasks.rados.rados.0.smithi170.stdout:584: oid 28 updating version 0 to 195 2024-03-27T09:37:41.129 INFO:tasks.rados.rados.0.smithi170.stdout:584: oid 28 version 195 is already newer than 195 2024-03-27T09:37:41.129 INFO:tasks.rados.rados.0.smithi170.stdout:584: oid 28 version 195 is already newer than 193 2024-03-27T09:37:41.129 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 195 (ObjNum 283 snap 76 seq_num 283) dirty exists 2024-03-27T09:37:41.129 INFO:tasks.rados.rados.0.smithi170.stdout:584: left oid 28 (ObjNum 283 snap 76 seq_num 283) 2024-03-27T09:37:42.038 INFO:tasks.rados.rados.0.smithi170.stdout:584: done (1 left) 2024-03-27T09:37:42.038 INFO:tasks.rados.rados.0.smithi170.stdout:585: done (0 left) 2024-03-27T09:37:42.038 INFO:tasks.rados.rados.0.smithi170.stdout:586: write (excl) oid 19 current snap is 77 2024-03-27T09:37:42.038 INFO:tasks.rados.rados.0.smithi170.stdout:586: seq_num 284 ranges {773188=475777,1969239=724762} 2024-03-27T09:37:42.044 INFO:tasks.rados.rados.0.smithi170.stdout:586: writing smithi17031214-19 from 773188 to 1248965 tid 1 2024-03-27T09:37:42.052 INFO:tasks.rados.rados.0.smithi170.stdout:586: writing smithi17031214-19 from 1969239 to 2694001 tid 2 2024-03-27T09:37:42.052 INFO:tasks.rados.rados.0.smithi170.stdout:587: snap_remove snap 72 2024-03-27T09:37:43.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:587: done (1 left) 2024-03-27T09:37:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:588: write (excl) oid 50 current snap is 77 2024-03-27T09:37:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:586: finishing write tid 1 to smithi17031214-19 2024-03-27T09:37:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:588: seq_num 285 ranges {789995=551026,2027721=468420} 2024-03-27T09:37:43.048 INFO:tasks.rados.rados.0.smithi170.stdout:588: writing smithi17031214-50 from 789995 to 1341021 tid 1 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:588: writing smithi17031214-50 from 2027721 to 2496141 tid 2 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:589: snap_create 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: finishing write tid 2 to smithi17031214-19 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: finishing write tid 3 to smithi17031214-19 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: finishing write tid 4 to smithi17031214-19 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: oid 19 updating version 0 to 130 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: oid 19 version 130 is already newer than 130 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: oid 19 version 130 is already newer than 128 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 130 (ObjNum 284 snap 77 seq_num 284) dirty exists 2024-03-27T09:37:43.056 INFO:tasks.rados.rados.0.smithi170.stdout:586: left oid 19 (ObjNum 284 snap 77 seq_num 284) 2024-03-27T09:37:43.156 INFO:tasks.rados.rados.0.smithi170.stdout:588: finishing write tid 1 to smithi17031214-50 2024-03-27T09:37:43.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:43.164+0000 7f6ad986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:43.172 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:43.171+0000 7f6ad986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:43.175 INFO:tasks.rados.rados.0.smithi170.stdout:588: finishing write tid 2 to smithi17031214-50 2024-03-27T09:37:43.182 INFO:tasks.rados.rados.0.smithi170.stdout:588: finishing write tid 3 to smithi17031214-50 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:588: finishing write tid 4 to smithi17031214-50 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:588: oid 50 updating version 0 to 231 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:588: oid 50 updating version 231 to 233 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:588: oid 50 version 233 is already newer than 233 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 233 (ObjNum 285 snap 77 seq_num 285) dirty exists 2024-03-27T09:37:43.188 INFO:tasks.rados.rados.0.smithi170.stdout:588: left oid 50 (ObjNum 285 snap 77 seq_num 285) 2024-03-27T09:37:43.198 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:43.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:43.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:43.685 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:43.688 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:44.044 INFO:tasks.rados.rados.0.smithi170.stdout:586: done (2 left) 2024-03-27T09:37:44.044 INFO:tasks.rados.rados.0.smithi170.stdout:588: done (1 left) 2024-03-27T09:37:44.044 INFO:tasks.rados.rados.0.smithi170.stdout:589: done (0 left) 2024-03-27T09:37:44.044 INFO:tasks.rados.rados.0.smithi170.stdout:590: write (excl) oid 2 current snap is 78 2024-03-27T09:37:44.044 INFO:tasks.rados.rados.0.smithi170.stdout:590: seq_num 286 ranges {406802=626275,1686203=728431} 2024-03-27T09:37:44.050 INFO:tasks.rados.rados.0.smithi170.stdout:590: writing smithi17031214-2 from 406802 to 1033077 tid 1 2024-03-27T09:37:44.058 INFO:tasks.rados.rados.0.smithi170.stdout:590: writing smithi17031214-2 from 1686203 to 2414634 tid 2 2024-03-27T09:37:44.058 INFO:tasks.rados.rados.0.smithi170.stdout:591: read oid 35 snap -1 2024-03-27T09:37:44.058 INFO:tasks.rados.rados.0.smithi170.stdout:591: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:37:44.058 INFO:tasks.rados.rados.0.smithi170.stdout:592: delete oid 8 current snap is 78 2024-03-27T09:37:44.099 INFO:tasks.rados.rados.0.smithi170.stdout:592: done (2 left) 2024-03-27T09:37:44.100 INFO:tasks.rados.rados.0.smithi170.stdout:593: write (excl) oid 32 current snap is 78 2024-03-27T09:37:44.100 INFO:tasks.rados.rados.0.smithi170.stdout:593: seq_num 288 ranges {440416=776773,1686814=732100} 2024-03-27T09:37:44.108 INFO:tasks.rados.rados.0.smithi170.stdout:593: writing smithi17031214-32 from 440416 to 1217189 tid 1 2024-03-27T09:37:44.115 INFO:tasks.rados.rados.0.smithi170.stdout:593: writing smithi17031214-32 from 1686814 to 2418914 tid 2 2024-03-27T09:37:44.116 INFO:tasks.rados.rados.0.smithi170.stdout:594: read oid 26 snap -1 2024-03-27T09:37:44.116 INFO:tasks.rados.rados.0.smithi170.stdout:594: expect (ObjNum 259 snap 70 seq_num 259) 2024-03-27T09:37:44.116 INFO:tasks.rados.rados.0.smithi170.stdout:595: write oid 30 current snap is 78 2024-03-27T09:37:44.116 INFO:tasks.rados.rados.0.smithi170.stdout:595: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-03-27T09:37:44.121 INFO:tasks.rados.rados.0.smithi170.stdout:595: writing smithi17031214-30 from 457223 to 1025598 tid 1 2024-03-27T09:37:44.129 INFO:tasks.rados.rados.0.smithi170.stdout:595: writing smithi17031214-30 from 1461649 to 1937407 tid 2 2024-03-27T09:37:44.136 INFO:tasks.rados.rados.0.smithi170.stdout:595: writing smithi17031214-30 from 2420186 to 2857223 tid 3 2024-03-27T09:37:44.136 INFO:tasks.rados.rados.0.smithi170.stdout:596: write (excl) oid 5 current snap is 78 2024-03-27T09:37:44.136 INFO:tasks.rados.rados.0.smithi170.stdout:596: seq_num 290 ranges {474030=643624,1520131=735769} 2024-03-27T09:37:44.145 INFO:tasks.rados.rados.0.smithi170.stdout:596: writing smithi17031214-5 from 474030 to 1117654 tid 1 2024-03-27T09:37:44.156 INFO:tasks.rados.rados.0.smithi170.stdout:596: writing smithi17031214-5 from 1520131 to 2255900 tid 2 2024-03-27T09:37:44.156 INFO:tasks.rados.rados.0.smithi170.stdout:597: write (excl) oid 6 current snap is 78 2024-03-27T09:37:44.156 INFO:tasks.rados.rados.0.smithi170.stdout:597: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-03-27T09:37:44.164 INFO:tasks.rados.rados.0.smithi170.stdout:597: writing smithi17031214-6 from 490837 to 1209710 tid 1 2024-03-27T09:37:44.174 INFO:tasks.rados.rados.0.smithi170.stdout:597: writing smithi17031214-6 from 1862260 to 2341687 tid 2 2024-03-27T09:37:44.176 INFO:tasks.rados.rados.0.smithi170.stdout:597: writing smithi17031214-6 from 2875032 to 2890837 tid 3 2024-03-27T09:37:44.176 INFO:tasks.rados.rados.0.smithi170.stdout:598: read oid 39 snap 76 2024-03-27T09:37:44.176 INFO:tasks.rados.rados.0.smithi170.stdout:598: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:44.176 INFO:tasks.rados.rados.0.smithi170.stdout:599: delete oid 21 current snap is 78 2024-03-27T09:37:44.176 INFO:tasks.rados.rados.0.smithi170.stdout:591: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-27T09:37:44.434 INFO:tasks.rados.rados.0.smithi170.stdout:590: finishing write tid 1 to smithi17031214-2 2024-03-27T09:37:44.434 INFO:tasks.rados.rados.0.smithi170.stdout:590: finishing write tid 2 to smithi17031214-2 2024-03-27T09:37:44.434 INFO:tasks.rados.rados.0.smithi170.stdout:591: done (8 left) 2024-03-27T09:37:44.434 INFO:tasks.rados.rados.0.smithi170.stdout:599: done (7 left) 2024-03-27T09:37:44.434 INFO:tasks.rados.rados.0.smithi170.stdout:600: snap_remove snap 71 2024-03-27T09:37:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:600: done (7 left) 2024-03-27T09:37:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:601: snap_create 2024-03-27T09:37:45.047 INFO:tasks.rados.rados.0.smithi170.stdout:590: finishing write tid 3 to smithi17031214-2 2024-03-27T09:37:45.048 INFO:tasks.rados.rados.0.smithi170.stdout:594: expect (ObjNum 259 snap 70 seq_num 259) 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:593: finishing write tid 1 to smithi17031214-32 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:590: finishing write tid 4 to smithi17031214-2 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:590: oid 2 updating version 0 to 207 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:590: oid 2 version 207 is already newer than 205 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:590: oid 2 version 207 is already newer than 206 2024-03-27T09:37:45.197 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 207 (ObjNum 286 snap 78 seq_num 286) dirty exists 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:590: left oid 2 (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:595: finishing write tid 1 to smithi17031214-30 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: finishing write tid 2 to smithi17031214-32 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:595: finishing write tid 2 to smithi17031214-30 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: finishing write tid 3 to smithi17031214-32 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: finishing write tid 4 to smithi17031214-32 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: oid 32 updating version 0 to 201 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: oid 32 version 201 is already newer than 199 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: oid 32 version 201 is already newer than 201 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 201 (ObjNum 288 snap 78 seq_num 288) dirty exists 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:593: left oid 32 (ObjNum 288 snap 78 seq_num 288) 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:596: finishing write tid 1 to smithi17031214-5 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:597: finishing write tid 1 to smithi17031214-6 2024-03-27T09:37:45.198 INFO:tasks.rados.rados.0.smithi170.stdout:595: finishing write tid 3 to smithi17031214-30 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:596: finishing write tid 2 to smithi17031214-5 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:597: finishing write tid 2 to smithi17031214-6 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: finishing write tid 4 to smithi17031214-30 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: finishing write tid 5 to smithi17031214-30 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: oid 30 updating version 0 to 164 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: oid 30 version 164 is already newer than 162 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: oid 30 version 164 is already newer than 163 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: oid 30 version 164 is already newer than 161 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 164 (ObjNum 289 snap 78 seq_num 289) dirty exists 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:595: left oid 30 (ObjNum 289 snap 78 seq_num 289) 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:597: finishing write tid 3 to smithi17031214-6 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:596: finishing write tid 3 to smithi17031214-5 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:596: finishing write tid 4 to smithi17031214-5 2024-03-27T09:37:45.199 INFO:tasks.rados.rados.0.smithi170.stdout:596: oid 5 updating version 0 to 99 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:596: oid 5 updating version 99 to 101 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:596: oid 5 version 101 is already newer than 100 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 101 (ObjNum 290 snap 78 seq_num 290) dirty exists 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:596: left oid 5 (ObjNum 290 snap 78 seq_num 290) 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: finishing write tid 4 to smithi17031214-6 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: finishing write tid 5 to smithi17031214-6 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: oid 6 updating version 0 to 115 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: oid 6 version 115 is already newer than 114 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: oid 6 version 115 is already newer than 112 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: oid 6 version 115 is already newer than 114 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 115 (ObjNum 291 snap 78 seq_num 291) dirty exists 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:597: left oid 6 (ObjNum 291 snap 78 seq_num 291) 2024-03-27T09:37:45.200 INFO:tasks.rados.rados.0.smithi170.stdout:598: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:45.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:45.354 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:45.352+0000 7f3a50024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:45.358 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:45.357+0000 7f3a50024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:45.387 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:45.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:590: done (7 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:593: done (6 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:594: done (5 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:595: done (4 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:596: done (3 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:597: done (2 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:598: done (1 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:601: done (0 left) 2024-03-27T09:37:45.960 INFO:tasks.rados.rados.0.smithi170.stdout:602: snap_create 2024-03-27T09:37:46.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~116s 2024-03-27T09:37:46.960 INFO:tasks.rados.rados.0.smithi170.stdout:602: done (0 left) 2024-03-27T09:37:46.961 INFO:tasks.rados.rados.0.smithi170.stdout:603: write oid 29 current snap is 80 2024-03-27T09:37:46.961 INFO:tasks.rados.rados.0.smithi170.stdout:603: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-03-27T09:37:46.966 INFO:tasks.rados.rados.0.smithi170.stdout:603: writing smithi17031214-29 from 524451 to 993822 tid 1 2024-03-27T09:37:46.972 INFO:tasks.rados.rados.0.smithi170.stdout:603: writing smithi17031214-29 from 1579224 to 2062320 tid 2 2024-03-27T09:37:46.976 INFO:tasks.rados.rados.0.smithi170.stdout:603: writing smithi17031214-29 from 2529878 to 2924451 tid 3 2024-03-27T09:37:46.976 INFO:tasks.rados.rados.0.smithi170.stdout:604: write (excl) oid 19 current snap is 80 2024-03-27T09:37:46.976 INFO:tasks.rados.rados.0.smithi170.stdout:604: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-03-27T09:37:46.982 INFO:tasks.rados.rados.0.smithi170.stdout:604: writing smithi17031214-19 from 541258 to 1085878 tid 1 2024-03-27T09:37:46.988 INFO:tasks.rados.rados.0.smithi170.stdout:604: writing smithi17031214-19 from 1637706 to 2264460 tid 2 2024-03-27T09:37:46.992 INFO:tasks.rados.rados.0.smithi170.stdout:604: writing smithi17031214-19 from 2840948 to 2941258 tid 3 2024-03-27T09:37:46.992 INFO:tasks.rados.rados.0.smithi170.stdout:605: write (excl) oid 31 current snap is 80 2024-03-27T09:37:46.992 INFO:tasks.rados.rados.0.smithi170.stdout:605: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-03-27T09:37:46.998 INFO:tasks.rados.rados.0.smithi170.stdout:605: writing smithi17031214-31 from 558065 to 1177934 tid 1 2024-03-27T09:37:47.003 INFO:tasks.rados.rados.0.smithi170.stdout:605: writing smithi17031214-31 from 1579835 to 2066600 tid 2 2024-03-27T09:37:47.008 INFO:tasks.rados.rados.0.smithi170.stdout:605: writing smithi17031214-31 from 2468371 to 2958065 tid 3 2024-03-27T09:37:47.008 INFO:tasks.rados.rados.0.smithi170.stdout:606: read oid 22 snap -1 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:606: expect (ObjNum 276 snap 72 seq_num 276) 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:607: read oid 12 snap 51 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:607: expect deleted 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:608: read oid 9 snap -1 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:608: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:609: write (excl) oid 24 current snap is 80 2024-03-27T09:37:47.009 INFO:tasks.rados.rados.0.smithi170.stdout:609: seq_num 296 ranges {574872=695118,2038317=630423} 2024-03-27T09:37:47.018 INFO:tasks.rados.rados.0.smithi170.stdout:609: writing smithi17031214-24 from 574872 to 1269990 tid 1 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:609: writing smithi17031214-24 from 2038317 to 2668740 tid 2 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:610: read oid 41 snap -1 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:610: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:611: read oid 17 snap -1 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:611: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:47.031 INFO:tasks.rados.rados.0.smithi170.stdout:612: delete oid 40 current snap is 80 2024-03-27T09:37:47.057 INFO:tasks.rados.rados.0.smithi170.stdout:603: finishing write tid 1 to smithi17031214-29 2024-03-27T09:37:47.079 INFO:tasks.rados.rados.0.smithi170.stdout:603: finishing write tid 2 to smithi17031214-29 2024-03-27T09:37:47.080 INFO:tasks.rados.rados.0.smithi170.stdout:610: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:37:47.201 INFO:tasks.rados.rados.0.smithi170.stdout:608: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:47.340 INFO:tasks.rados.rados.0.smithi170.stdout:604: finishing write tid 1 to smithi17031214-19 2024-03-27T09:37:47.340 INFO:tasks.rados.rados.0.smithi170.stdout:603: finishing write tid 3 to smithi17031214-29 2024-03-27T09:37:47.340 INFO:tasks.rados.rados.0.smithi170.stdout:606: expect (ObjNum 276 snap 72 seq_num 276) 2024-03-27T09:37:47.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:47.497 INFO:tasks.rados.rados.0.smithi170.stdout:604: finishing write tid 2 to smithi17031214-19 2024-03-27T09:37:47.497 INFO:tasks.rados.rados.0.smithi170.stdout:606: done (9 left) 2024-03-27T09:37:47.497 INFO:tasks.rados.rados.0.smithi170.stdout:608: done (8 left) 2024-03-27T09:37:47.497 INFO:tasks.rados.rados.0.smithi170.stdout:610: done (7 left) 2024-03-27T09:37:47.498 INFO:tasks.rados.rados.0.smithi170.stdout:612: done (6 left) 2024-03-27T09:37:47.498 INFO:tasks.rados.rados.0.smithi170.stdout:613: read oid 16 snap 67 2024-03-27T09:37:47.498 INFO:tasks.rados.rados.0.smithi170.stdout:613: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:47.498 INFO:tasks.rados.rados.0.smithi170.stdout:614: write (excl) oid 42 current snap is 80 2024-03-27T09:37:47.498 INFO:tasks.rados.rados.0.smithi170.stdout:614: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-03-27T09:37:47.503 INFO:tasks.rados.rados.0.smithi170.stdout:614: writing smithi17031214-42 from 608486 to 1170455 tid 1 2024-03-27T09:37:47.511 INFO:tasks.rados.rados.0.smithi170.stdout:614: writing smithi17031214-42 from 1871634 to 2505726 tid 2 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:614: writing smithi17031214-42 from 2950640 to 3008486 tid 3 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:615: delete oid 43 current snap is 80 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:603: finishing write tid 4 to smithi17031214-29 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:603: finishing write tid 5 to smithi17031214-29 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:603: oid 29 updating version 0 to 198 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:603: oid 29 version 198 is already newer than 195 2024-03-27T09:37:47.515 INFO:tasks.rados.rados.0.smithi170.stdout:603: oid 29 version 198 is already newer than 198 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:603: oid 29 version 198 is already newer than 196 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 198 (ObjNum 293 snap 80 seq_num 293) dirty exists 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:603: left oid 29 (ObjNum 293 snap 80 seq_num 293) 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:609: finishing write tid 1 to smithi17031214-24 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:604: finishing write tid 3 to smithi17031214-19 2024-03-27T09:37:47.516 INFO:tasks.rados.rados.0.smithi170.stdout:611: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:47.547 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:47.544+0000 7ff27d514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:47.554 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:47.550+0000 7ff27d514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:47.576 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:47.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: finishing write tid 4 to smithi17031214-19 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: finishing write tid 5 to smithi17031214-19 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: oid 19 updating version 0 to 141 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: oid 19 version 141 is already newer than 140 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: oid 19 updating version 141 to 143 2024-03-27T09:37:47.834 INFO:tasks.rados.rados.0.smithi170.stdout:604: oid 19 version 143 is already newer than 143 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 143 (ObjNum 294 snap 80 seq_num 294) dirty exists 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:604: left oid 19 (ObjNum 294 snap 80 seq_num 294) 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:605: finishing write tid 1 to smithi17031214-31 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:603: done (8 left) 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:604: done (7 left) 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:611: done (6 left) 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:615: done (5 left) 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:616: write oid 37 current snap is 80 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:609: finishing write tid 2 to smithi17031214-24 2024-03-27T09:37:47.835 INFO:tasks.rados.rados.0.smithi170.stdout:616: seq_num 300 ranges {642100=712467,1872245=637761} 2024-03-27T09:37:47.841 INFO:tasks.rados.rados.0.smithi170.stdout:616: writing smithi17031214-37 from 642100 to 1354567 tid 1 2024-03-27T09:37:47.848 INFO:tasks.rados.rados.0.smithi170.stdout:616: writing smithi17031214-37 from 1872245 to 2510006 tid 2 2024-03-27T09:37:47.848 INFO:tasks.rados.rados.0.smithi170.stdout:617: write oid 29 current snap is 80 2024-03-27T09:37:47.848 INFO:tasks.rados.rados.0.smithi170.stdout:609: finishing write tid 3 to smithi17031214-24 2024-03-27T09:37:47.848 INFO:tasks.rados.rados.0.smithi170.stdout:617: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-03-27T09:37:47.856 INFO:tasks.rados.rados.0.smithi170.stdout:617: writing smithi17031214-29 from 658907 to 1446623 tid 1 2024-03-27T09:37:47.862 INFO:tasks.rados.rados.0.smithi170.stdout:617: writing smithi17031214-29 from 1930727 to 2428499 tid 2 2024-03-27T09:37:47.866 INFO:tasks.rados.rados.0.smithi170.stdout:617: writing smithi17031214-29 from 3032909 to 3058907 tid 3 2024-03-27T09:37:47.866 INFO:tasks.rados.rados.0.smithi170.stdout:618: delete oid 8 current snap is 80 2024-03-27T09:37:47.866 INFO:tasks.rados.rados.0.smithi170.stdout:609: finishing write tid 4 to smithi17031214-24 2024-03-27T09:37:47.866 INFO:tasks.rados.rados.0.smithi170.stdout:609: oid 24 updating version 0 to 90 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:609: oid 24 updating version 90 to 91 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:609: oid 24 updating version 91 to 92 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 92 (ObjNum 296 snap 80 seq_num 296) dirty exists 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:609: left oid 24 (ObjNum 296 snap 80 seq_num 296) 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: finishing write tid 2 to smithi17031214-31 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: finishing write tid 3 to smithi17031214-31 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: finishing write tid 4 to smithi17031214-31 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: finishing write tid 5 to smithi17031214-31 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: oid 31 updating version 0 to 201 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: oid 31 updating version 201 to 203 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: oid 31 version 203 is already newer than 202 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: oid 31 version 203 is already newer than 200 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 203 (ObjNum 295 snap 80 seq_num 295) dirty exists 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:605: left oid 31 (ObjNum 295 snap 80 seq_num 295) 2024-03-27T09:37:47.867 INFO:tasks.rados.rados.0.smithi170.stdout:613: expect (ObjNum 176 snap 33 seq_num 176) 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: finishing write tid 1 to smithi17031214-42 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: finishing write tid 2 to smithi17031214-42 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: finishing write tid 3 to smithi17031214-42 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: finishing write tid 4 to smithi17031214-42 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: finishing write tid 5 to smithi17031214-42 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: oid 42 updating version 0 to 265 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: oid 42 updating version 265 to 267 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: oid 42 version 267 is already newer than 264 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:614: oid 42 version 267 is already newer than 267 2024-03-27T09:37:47.980 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 267 (ObjNum 298 snap 80 seq_num 298) dirty exists 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:614: left oid 42 (ObjNum 298 snap 80 seq_num 298) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: finishing write tid 1 to smithi17031214-37 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:605: done (7 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:607: done (6 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:609: done (5 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:613: done (4 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:614: done (3 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:618: done (2 left) 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:619: write oid 7 current snap is 80 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: finishing write tid 2 to smithi17031214-37 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:617: finishing write tid 1 to smithi17031214-29 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: finishing write tid 3 to smithi17031214-37 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: finishing write tid 4 to smithi17031214-37 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: oid 37 updating version 0 to 206 2024-03-27T09:37:47.981 INFO:tasks.rados.rados.0.smithi170.stdout:616: oid 37 updating version 206 to 208 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:616: oid 37 version 208 is already newer than 207 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 208 (ObjNum 300 snap 80 seq_num 300) dirty exists 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:616: left oid 37 (ObjNum 300 snap 80 seq_num 300) 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: finishing write tid 2 to smithi17031214-29 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: finishing write tid 3 to smithi17031214-29 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: finishing write tid 4 to smithi17031214-29 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: finishing write tid 5 to smithi17031214-29 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: oid 29 updating version 0 to 205 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: oid 29 updating version 205 to 206 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: oid 29 updating version 206 to 207 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: oid 29 version 207 is already newer than 204 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 207 (ObjNum 301 snap 80 seq_num 301) dirty exists 2024-03-27T09:37:47.982 INFO:tasks.rados.rados.0.smithi170.stdout:617: left oid 29 (ObjNum 301 snap 80 seq_num 301) 2024-03-27T09:37:47.983 INFO:tasks.rados.rados.0.smithi170.stdout:619: seq_num 303 ranges {692521=538214,1931338=785088} 2024-03-27T09:37:47.985 INFO:tasks.rados.rados.0.smithi170.stdout:619: writing smithi17031214-7 from 692521 to 1230735 tid 1 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:619: writing smithi17031214-7 from 1931338 to 2716426 tid 2 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:616: done (2 left) 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:617: done (1 left) 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:620: read oid 2 snap -1 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:620: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:621: write (excl) oid 1 current snap is 80 2024-03-27T09:37:47.994 INFO:tasks.rados.rados.0.smithi170.stdout:621: seq_num 304 ranges {709328=613463,1989820=645099} 2024-03-27T09:37:48.000 INFO:tasks.rados.rados.0.smithi170.stdout:621: writing smithi17031214-1 from 709328 to 1322791 tid 1 2024-03-27T09:37:48.008 INFO:tasks.rados.rados.0.smithi170.stdout:621: writing smithi17031214-1 from 1989820 to 2634919 tid 2 2024-03-27T09:37:48.008 INFO:tasks.rados.rados.0.smithi170.stdout:622: read oid 11 snap 78 2024-03-27T09:37:48.008 INFO:tasks.rados.rados.0.smithi170.stdout:622: expect (ObjNum 215 snap 46 seq_num 215) 2024-03-27T09:37:48.008 INFO:tasks.rados.rados.0.smithi170.stdout:623: delete oid 6 current snap is 80 2024-03-27T09:37:48.053 INFO:tasks.rados.rados.0.smithi170.stdout:619: finishing write tid 1 to smithi17031214-7 2024-03-27T09:37:48.091 INFO:tasks.rados.rados.0.smithi170.stdout:619: finishing write tid 2 to smithi17031214-7 2024-03-27T09:37:48.091 INFO:tasks.rados.rados.0.smithi170.stdout:621: finishing write tid 1 to smithi17031214-1 2024-03-27T09:37:48.098 INFO:tasks.rados.rados.0.smithi170.stdout:619: finishing write tid 3 to smithi17031214-7 2024-03-27T09:37:48.100 INFO:tasks.rados.rados.0.smithi170.stdout:619: finishing write tid 4 to smithi17031214-7 2024-03-27T09:37:48.100 INFO:tasks.rados.rados.0.smithi170.stdout:619: oid 7 updating version 0 to 60 2024-03-27T09:37:48.100 INFO:tasks.rados.rados.0.smithi170.stdout:619: oid 7 version 60 is already newer than 58 2024-03-27T09:37:48.101 INFO:tasks.rados.rados.0.smithi170.stdout:619: oid 7 version 60 is already newer than 60 2024-03-27T09:37:48.101 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 60 (ObjNum 303 snap 80 seq_num 303) dirty exists 2024-03-27T09:37:48.101 INFO:tasks.rados.rados.0.smithi170.stdout:619: left oid 7 (ObjNum 303 snap 80 seq_num 303) 2024-03-27T09:37:48.119 INFO:tasks.rados.rados.0.smithi170.stdout:620: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:37:48.225 INFO:tasks.rados.rados.0.smithi170.stdout:621: finishing write tid 2 to smithi17031214-1 2024-03-27T09:37:48.225 INFO:tasks.rados.rados.0.smithi170.stdout:622: expect (ObjNum 215 snap 46 seq_num 215) 2024-03-27T09:37:48.358 INFO:tasks.rados.rados.0.smithi170.stdout:621: finishing write tid 3 to smithi17031214-1 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: finishing write tid 4 to smithi17031214-1 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: oid 1 updating version 0 to 123 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: oid 1 version 123 is already newer than 121 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: oid 1 version 123 is already newer than 122 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 123 (ObjNum 304 snap 80 seq_num 304) dirty exists 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: left oid 1 (ObjNum 304 snap 80 seq_num 304) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:619: done (4 left) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:620: done (3 left) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:621: done (2 left) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:622: done (1 left) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:623: done (0 left) 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:624: write oid 35 current snap is 80 2024-03-27T09:37:48.359 INFO:tasks.rados.rados.0.smithi170.stdout:624: seq_num 306 ranges {742942=480314,1823137=648768} 2024-03-27T09:37:48.364 INFO:tasks.rados.rados.0.smithi170.stdout:624: writing smithi17031214-35 from 742942 to 1223256 tid 1 2024-03-27T09:37:48.372 INFO:tasks.rados.rados.0.smithi170.stdout:624: writing smithi17031214-35 from 1823137 to 2471905 tid 2 2024-03-27T09:37:48.372 INFO:tasks.rados.rados.0.smithi170.stdout:625: read oid 40 snap -1 2024-03-27T09:37:48.372 INFO:tasks.rados.rados.0.smithi170.stdout:625: expect deleted 2024-03-27T09:37:48.372 INFO:tasks.rados.rados.0.smithi170.stdout:626: snap_create 2024-03-27T09:37:48.395 INFO:tasks.rados.rados.0.smithi170.stdout:624: finishing write tid 1 to smithi17031214-35 2024-03-27T09:37:48.411 INFO:tasks.rados.rados.0.smithi170.stdout:624: finishing write tid 2 to smithi17031214-35 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: finishing write tid 3 to smithi17031214-35 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: finishing write tid 4 to smithi17031214-35 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: oid 35 updating version 0 to 211 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: oid 35 updating version 211 to 212 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: oid 35 version 212 is already newer than 210 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 212 (ObjNum 306 snap 80 seq_num 306) dirty exists 2024-03-27T09:37:48.430 INFO:tasks.rados.rados.0.smithi170.stdout:624: left oid 35 (ObjNum 306 snap 80 seq_num 306) 2024-03-27T09:37:48.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:48.789 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:48.792 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:49.057 INFO:tasks.rados.rados.0.smithi170.stdout:624: done (2 left) 2024-03-27T09:37:49.057 INFO:tasks.rados.rados.0.smithi170.stdout:625: done (1 left) 2024-03-27T09:37:49.057 INFO:tasks.rados.rados.0.smithi170.stdout:626: done (0 left) 2024-03-27T09:37:49.057 INFO:tasks.rados.rados.0.smithi170.stdout:627: delete oid 4 current snap is 81 2024-03-27T09:37:49.110 INFO:tasks.rados.rados.0.smithi170.stdout:627: done (0 left) 2024-03-27T09:37:49.111 INFO:tasks.rados.rados.0.smithi170.stdout:628: delete oid 1 current snap is 81 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:628: done (0 left) 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:629: read oid 41 snap 51 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:630: read oid 28 snap -1 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:630: expect (ObjNum 283 snap 76 seq_num 283) 2024-03-27T09:37:49.177 INFO:tasks.rados.rados.0.smithi170.stdout:631: snap_remove snap 67 2024-03-27T09:37:49.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:49.737 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:49.735+0000 7f9aa686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:49.741 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:49.740+0000 7f9aa686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:49.765 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:49.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:50.061 INFO:tasks.rados.rados.0.smithi170.stdout:631: done (2 left) 2024-03-27T09:37:50.061 INFO:tasks.rados.rados.0.smithi170.stdout:632: delete oid 4 current snap is 81 2024-03-27T09:37:50.061 INFO:tasks.rados.rados.0.smithi170.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-27T09:37:50.302 INFO:tasks.rados.rados.0.smithi170.stdout:630: expect (ObjNum 283 snap 76 seq_num 283) 2024-03-27T09:37:50.524 INFO:tasks.rados.rados.0.smithi170.stdout:629: done (2 left) 2024-03-27T09:37:50.524 INFO:tasks.rados.rados.0.smithi170.stdout:630: done (1 left) 2024-03-27T09:37:50.524 INFO:tasks.rados.rados.0.smithi170.stdout:632: done (0 left) 2024-03-27T09:37:50.524 INFO:tasks.rados.rados.0.smithi170.stdout:633: write oid 44 current snap is 81 2024-03-27T09:37:50.524 INFO:tasks.rados.rados.0.smithi170.stdout:633: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-03-27T09:37:50.529 INFO:tasks.rados.rados.0.smithi170.stdout:633: writing smithi17031214-44 from 410170 to 1191480 tid 1 2024-03-27T09:37:50.537 INFO:tasks.rados.rados.0.smithi170.stdout:633: writing smithi17031214-44 from 1940712 to 2596818 tid 2 2024-03-27T09:37:50.540 INFO:tasks.rados.rados.0.smithi170.stdout:633: writing smithi17031214-44 from 3163363 to 3210170 tid 3 2024-03-27T09:37:50.541 INFO:tasks.rados.rados.0.smithi170.stdout:634: delete oid 19 current snap is 81 2024-03-27T09:37:50.570 INFO:tasks.rados.rados.0.smithi170.stdout:633: finishing write tid 1 to smithi17031214-44 2024-03-27T09:37:50.589 INFO:tasks.rados.rados.0.smithi170.stdout:634: done (1 left) 2024-03-27T09:37:50.589 INFO:tasks.rados.rados.0.smithi170.stdout:635: read oid 42 snap -1 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:635: expect (ObjNum 298 snap 80 seq_num 298) 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:636: read oid 13 snap -1 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:636: expect deleted 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:637: read oid 15 snap -1 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:637: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:638: read oid 22 snap -1 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:638: expect (ObjNum 276 snap 72 seq_num 276) 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:639: write (excl) oid 32 current snap is 81 2024-03-27T09:37:50.590 INFO:tasks.rados.rados.0.smithi170.stdout:639: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-03-27T09:37:50.597 INFO:tasks.rados.rados.0.smithi170.stdout:639: writing smithi17031214-32 from 443784 to 1091945 tid 1 2024-03-27T09:37:50.607 INFO:tasks.rados.rados.0.smithi170.stdout:639: writing smithi17031214-32 from 1657676 to 2317451 tid 2 2024-03-27T09:37:50.614 INFO:tasks.rados.rados.0.smithi170.stdout:639: writing smithi17031214-32 from 2818209 to 3243784 tid 3 2024-03-27T09:37:50.614 INFO:tasks.rados.rados.0.smithi170.stdout:640: delete oid 31 current snap is 81 2024-03-27T09:37:50.615 INFO:tasks.rados.rados.0.smithi170.stdout:633: finishing write tid 2 to smithi17031214-44 2024-03-27T09:37:50.615 INFO:tasks.rados.rados.0.smithi170.stdout:633: finishing write tid 3 to smithi17031214-44 2024-03-27T09:37:50.620 INFO:tasks.rados.rados.0.smithi170.stdout:633: finishing write tid 4 to smithi17031214-44 2024-03-27T09:37:50.624 INFO:tasks.rados.rados.0.smithi170.stdout:633: finishing write tid 5 to smithi17031214-44 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:633: oid 44 updating version 0 to 244 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:633: oid 44 version 244 is already newer than 243 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:633: oid 44 updating version 244 to 246 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:633: oid 44 version 246 is already newer than 246 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 246 (ObjNum 310 snap 81 seq_num 310) dirty exists 2024-03-27T09:37:50.625 INFO:tasks.rados.rados.0.smithi170.stdout:633: left oid 44 (ObjNum 310 snap 81 seq_num 310) 2024-03-27T09:37:50.629 INFO:tasks.rados.rados.0.smithi170.stdout:635: expect (ObjNum 298 snap 80 seq_num 298) 2024-03-27T09:37:50.765 INFO:tasks.rados.rados.0.smithi170.stdout:639: finishing write tid 1 to smithi17031214-32 2024-03-27T09:37:50.765 INFO:tasks.rados.rados.0.smithi170.stdout:637: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-27T09:37:50.888 INFO:tasks.rados.rados.0.smithi170.stdout:638: expect (ObjNum 276 snap 72 seq_num 276) 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: finishing write tid 2 to smithi17031214-32 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: finishing write tid 3 to smithi17031214-32 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: finishing write tid 4 to smithi17031214-32 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: finishing write tid 5 to smithi17031214-32 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: oid 32 updating version 0 to 225 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: oid 32 version 225 is already newer than 222 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: oid 32 version 225 is already newer than 223 2024-03-27T09:37:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:639: oid 32 version 225 is already newer than 225 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 225 (ObjNum 312 snap 81 seq_num 312) dirty exists 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:639: left oid 32 (ObjNum 312 snap 81 seq_num 312) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:633: done (6 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:635: done (5 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:636: done (4 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:637: done (3 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:638: done (2 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:639: done (1 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:640: done (0 left) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:641: read oid 3 snap 63 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:641: expect (ObjNum 169 snap 26 seq_num 169) 2024-03-27T09:37:51.051 INFO:tasks.rados.rados.0.smithi170.stdout:642: delete oid 49 current snap is 81 2024-03-27T09:37:51.074 INFO:tasks.rados.rados.0.smithi170.stdout:641: expect (ObjNum 169 snap 26 seq_num 169) 2024-03-27T09:37:51.234 INFO:tasks.rados.rados.0.smithi170.stdout:641: done (1 left) 2024-03-27T09:37:51.234 INFO:tasks.rados.rados.0.smithi170.stdout:642: done (0 left) 2024-03-27T09:37:51.234 INFO:tasks.rados.rados.0.smithi170.stdout:643: snap_remove snap 75 2024-03-27T09:37:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~121s 2024-03-27T09:37:51.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:51.924 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:51.921+0000 7fa4d926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:51.927 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:51.927+0000 7fa4d926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:51.952 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:52.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:52.064 INFO:tasks.rados.rados.0.smithi170.stdout:643: done (0 left) 2024-03-27T09:37:52.064 INFO:tasks.rados.rados.0.smithi170.stdout:644: snap_remove snap 73 2024-03-27T09:37:53.068 INFO:tasks.rados.rados.0.smithi170.stdout:644: done (0 left) 2024-03-27T09:37:53.068 INFO:tasks.rados.rados.0.smithi170.stdout:645: read oid 39 snap -1 2024-03-27T09:37:53.069 INFO:tasks.rados.rados.0.smithi170.stdout:645: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:53.069 INFO:tasks.rados.rados.0.smithi170.stdout:646: write oid 45 current snap is 81 2024-03-27T09:37:53.069 INFO:tasks.rados.rados.0.smithi170.stdout:646: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-03-27T09:37:53.074 INFO:tasks.rados.rados.0.smithi170.stdout:646: writing smithi17031214-45 from 494205 to 968113 tid 1 2024-03-27T09:37:53.078 INFO:tasks.rados.rados.0.smithi170.stdout:646: writing smithi17031214-45 from 1433122 to 1840224 tid 2 2024-03-27T09:37:53.085 INFO:tasks.rados.rados.0.smithi170.stdout:646: writing smithi17031214-45 from 2384125 to 2846809 tid 3 2024-03-27T09:37:53.085 INFO:tasks.rados.rados.0.smithi170.stdout:647: delete oid 27 current snap is 81 2024-03-27T09:37:53.135 INFO:tasks.rados.rados.0.smithi170.stdout:645: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: finishing write tid 1 to smithi17031214-45 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: finishing write tid 2 to smithi17031214-45 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: finishing write tid 3 to smithi17031214-45 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: finishing write tid 4 to smithi17031214-45 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: finishing write tid 5 to smithi17031214-45 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: oid 45 updating version 0 to 222 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: oid 45 version 222 is already newer than 221 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: oid 45 version 222 is already newer than 219 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:646: oid 45 version 222 is already newer than 221 2024-03-27T09:37:53.337 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 222 (ObjNum 315 snap 81 seq_num 315) dirty exists 2024-03-27T09:37:53.338 INFO:tasks.rados.rados.0.smithi170.stdout:646: left oid 45 (ObjNum 315 snap 81 seq_num 315) 2024-03-27T09:37:53.338 INFO:tasks.rados.rados.0.smithi170.stdout:645: done (2 left) 2024-03-27T09:37:53.338 INFO:tasks.rados.rados.0.smithi170.stdout:646: done (1 left) 2024-03-27T09:37:53.338 INFO:tasks.rados.rados.0.smithi170.stdout:647: done (0 left) 2024-03-27T09:37:53.338 INFO:tasks.rados.rados.0.smithi170.stdout:648: snap_remove snap 79 2024-03-27T09:37:53.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:53.896 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:53.899 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:54.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:54.071 INFO:tasks.rados.rados.0.smithi170.stdout:648: done (0 left) 2024-03-27T09:37:54.071 INFO:tasks.rados.rados.0.smithi170.stdout:649: write oid 19 current snap is 81 2024-03-27T09:37:54.071 INFO:tasks.rados.rados.0.smithi170.stdout:649: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-03-27T09:37:54.077 INFO:tasks.rados.rados.0.smithi170.stdout:649: writing smithi17031214-19 from 527819 to 1152225 tid 1 2024-03-27T09:37:54.083 INFO:tasks.rados.rados.0.smithi170.stdout:649: writing smithi17031214-19 from 1833733 to 2244504 tid 2 2024-03-27T09:37:54.089 INFO:tasks.rados.rados.0.smithi170.stdout:649: writing smithi17031214-19 from 2722618 to 3324199 tid 3 2024-03-27T09:37:54.089 INFO:tasks.rados.rados.0.smithi170.stdout:650: delete oid 25 current snap is 81 2024-03-27T09:37:54.112 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:54.110+0000 7f18720ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:54.115 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:54.115+0000 7f18720ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:54.142 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:54.223 INFO:tasks.rados.rados.0.smithi170.stdout:649: finishing write tid 1 to smithi17031214-19 2024-03-27T09:37:54.234 INFO:tasks.rados.rados.0.smithi170.stdout:650: done (1 left) 2024-03-27T09:37:54.234 INFO:tasks.rados.rados.0.smithi170.stdout:651: read oid 17 snap -1 2024-03-27T09:37:54.234 INFO:tasks.rados.rados.0.smithi170.stdout:651: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:54.234 INFO:tasks.rados.rados.0.smithi170.stdout:652: delete oid 13 current snap is 81 2024-03-27T09:37:54.248 INFO:tasks.rados.rados.0.smithi170.stdout:649: finishing write tid 2 to smithi17031214-19 2024-03-27T09:37:54.283 INFO:tasks.rados.rados.0.smithi170.stdout:652: done (2 left) 2024-03-27T09:37:54.283 INFO:tasks.rados.rados.0.smithi170.stdout:653: read oid 12 snap -1 2024-03-27T09:37:54.283 INFO:tasks.rados.rados.0.smithi170.stdout:653: expect deleted 2024-03-27T09:37:54.283 INFO:tasks.rados.rados.0.smithi170.stdout:654: delete oid 13 current snap is 81 2024-03-27T09:37:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:54.284 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:54.288 INFO:tasks.rados.rados.0.smithi170.stdout:654: done (3 left) 2024-03-27T09:37:54.288 INFO:tasks.rados.rados.0.smithi170.stdout:655: read oid 25 snap -1 2024-03-27T09:37:54.288 INFO:tasks.rados.rados.0.smithi170.stdout:655: expect deleted 2024-03-27T09:37:54.288 INFO:tasks.rados.rados.0.smithi170.stdout:656: snap_remove snap 63 2024-03-27T09:37:55.074 INFO:tasks.rados.rados.0.smithi170.stdout:656: done (4 left) 2024-03-27T09:37:55.074 INFO:tasks.rados.rados.0.smithi170.stdout:657: read oid 40 snap -1 2024-03-27T09:37:55.074 INFO:tasks.rados.rados.0.smithi170.stdout:657: expect deleted 2024-03-27T09:37:55.074 INFO:tasks.rados.rados.0.smithi170.stdout:658: write oid 11 current snap is 81 2024-03-27T09:37:55.074 INFO:tasks.rados.rados.0.smithi170.stdout:658: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-03-27T09:37:55.081 INFO:tasks.rados.rados.0.smithi170.stdout:658: writing smithi17031214-11 from 595047 to 1236802 tid 1 2024-03-27T09:37:55.086 INFO:tasks.rados.rados.0.smithi170.stdout:658: writing smithi17031214-11 from 1667661 to 2085770 tid 2 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:658: writing smithi17031214-11 from 2548663 to 3311685 tid 3 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:659: read oid 39 snap -1 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:659: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:660: read oid 49 snap -1 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:660: expect deleted 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:661: write (excl) oid 47 current snap is 81 2024-03-27T09:37:55.094 INFO:tasks.rados.rados.0.smithi170.stdout:661: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-03-27T09:37:55.104 INFO:tasks.rados.rados.0.smithi170.stdout:661: writing smithi17031214-47 from 611854 to 1328858 tid 1 2024-03-27T09:37:55.114 INFO:tasks.rados.rados.0.smithi170.stdout:661: writing smithi17031214-47 from 2126143 to 2687910 tid 2 2024-03-27T09:37:55.119 INFO:tasks.rados.rados.0.smithi170.stdout:661: writing smithi17031214-47 from 3259733 to 3411854 tid 3 2024-03-27T09:37:55.119 INFO:tasks.rados.rados.0.smithi170.stdout:662: read oid 14 snap -1 2024-03-27T09:37:55.119 INFO:tasks.rados.rados.0.smithi170.stdout:662: expect (ObjNum 232 snap 55 seq_num 232) 2024-03-27T09:37:55.120 INFO:tasks.rados.rados.0.smithi170.stdout:663: write (excl) oid 26 current snap is 81 2024-03-27T09:37:55.120 INFO:tasks.rados.rados.0.smithi170.stdout:649: finishing write tid 3 to smithi17031214-19 2024-03-27T09:37:55.120 INFO:tasks.rados.rados.0.smithi170.stdout:651: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: finishing write tid 4 to smithi17031214-19 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: finishing write tid 5 to smithi17031214-19 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: oid 19 updating version 0 to 162 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: oid 19 updating version 162 to 165 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: oid 19 version 165 is already newer than 163 2024-03-27T09:37:55.482 INFO:tasks.rados.rados.0.smithi170.stdout:649: oid 19 version 165 is already newer than 164 2024-03-27T09:37:55.483 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 165 (ObjNum 317 snap 81 seq_num 317) dirty exists 2024-03-27T09:37:55.483 INFO:tasks.rados.rados.0.smithi170.stdout:649: left oid 19 (ObjNum 317 snap 81 seq_num 317) 2024-03-27T09:37:55.483 INFO:tasks.rados.rados.0.smithi170.stdout:663: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-03-27T09:37:55.489 INFO:tasks.rados.rados.0.smithi170.stdout:663: writing smithi17031214-26 from 628661 to 1420914 tid 1 2024-03-27T09:37:55.496 INFO:tasks.rados.rados.0.smithi170.stdout:663: writing smithi17031214-26 from 2184625 to 2606403 tid 2 2024-03-27T09:37:55.500 INFO:tasks.rados.rados.0.smithi170.stdout:663: writing smithi17031214-26 from 3403509 to 3428661 tid 3 2024-03-27T09:37:55.501 INFO:tasks.rados.rados.0.smithi170.stdout:649: done (10 left) 2024-03-27T09:37:55.501 INFO:tasks.rados.rados.0.smithi170.stdout:651: done (9 left) 2024-03-27T09:37:55.501 INFO:tasks.rados.rados.0.smithi170.stdout:653: done (8 left) 2024-03-27T09:37:55.501 INFO:tasks.rados.rados.0.smithi170.stdout:655: done (7 left) 2024-03-27T09:37:55.501 INFO:tasks.rados.rados.0.smithi170.stdout:664: snap_remove snap 76 2024-03-27T09:37:56.077 INFO:tasks.rados.rados.0.smithi170.stdout:664: done (7 left) 2024-03-27T09:37:56.077 INFO:tasks.rados.rados.0.smithi170.stdout:665: delete oid 6 current snap is 81 2024-03-27T09:37:56.077 INFO:tasks.rados.rados.0.smithi170.stdout:662: expect (ObjNum 232 snap 55 seq_num 232) 2024-03-27T09:37:56.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:37:56.394 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:56.392+0000 7fcd87ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:56.398 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:56.397+0000 7fcd87ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:56.431 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:56.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:56.483 INFO:tasks.rados.rados.0.smithi170.stdout:659: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:661: finishing write tid 1 to smithi17031214-47 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: finishing write tid 1 to smithi17031214-11 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: finishing write tid 2 to smithi17031214-11 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: finishing write tid 3 to smithi17031214-11 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: finishing write tid 4 to smithi17031214-11 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:661: finishing write tid 2 to smithi17031214-47 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: finishing write tid 5 to smithi17031214-11 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: oid 11 updating version 0 to 234 2024-03-27T09:37:56.681 INFO:tasks.rados.rados.0.smithi170.stdout:658: oid 11 version 234 is already newer than 234 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:658: oid 11 version 234 is already newer than 232 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:658: oid 11 version 234 is already newer than 232 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 234 (ObjNum 321 snap 81 seq_num 321) dirty exists 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:658: left oid 11 (ObjNum 321 snap 81 seq_num 321) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:661: finishing write tid 3 to smithi17031214-47 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:657: done (7 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:658: done (6 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:659: done (5 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:660: done (4 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:662: done (3 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:665: done (2 left) 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:666: delete oid 38 current snap is 81 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:661: finishing write tid 4 to smithi17031214-47 2024-03-27T09:37:56.682 INFO:tasks.rados.rados.0.smithi170.stdout:661: finishing write tid 5 to smithi17031214-47 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:661: oid 47 updating version 0 to 116 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:661: oid 47 version 116 is already newer than 113 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:661: oid 47 version 116 is already newer than 114 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:661: oid 47 version 116 is already newer than 115 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 116 (ObjNum 322 snap 81 seq_num 322) dirty exists 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:661: left oid 47 (ObjNum 322 snap 81 seq_num 322) 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: finishing write tid 1 to smithi17031214-26 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: finishing write tid 2 to smithi17031214-26 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: finishing write tid 3 to smithi17031214-26 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: finishing write tid 4 to smithi17031214-26 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: finishing write tid 5 to smithi17031214-26 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: oid 26 updating version 0 to 175 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: oid 26 version 175 is already newer than 173 2024-03-27T09:37:56.683 INFO:tasks.rados.rados.0.smithi170.stdout:663: oid 26 version 175 is already newer than 174 2024-03-27T09:37:56.684 INFO:tasks.rados.rados.0.smithi170.stdout:663: oid 26 updating version 175 to 176 2024-03-27T09:37:56.684 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 176 (ObjNum 323 snap 81 seq_num 323) dirty exists 2024-03-27T09:37:56.684 INFO:tasks.rados.rados.0.smithi170.stdout:663: left oid 26 (ObjNum 323 snap 81 seq_num 323) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:661: done (2 left) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:663: done (1 left) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:666: done (0 left) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:667: read oid 17 snap -1 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:667: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:668: read oid 6 snap -1 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:668: expect deleted 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:669: read oid 42 snap -1 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:669: expect (ObjNum 298 snap 80 seq_num 298) 2024-03-27T09:37:56.821 INFO:tasks.rados.rados.0.smithi170.stdout:670: delete oid 14 current snap is 81 2024-03-27T09:37:56.878 INFO:tasks.rados.rados.0.smithi170.stdout:667: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:37:57.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2024-03-27T09:37:57.198 INFO:tasks.rados.rados.0.smithi170.stdout:669: expect (ObjNum 298 snap 80 seq_num 298) 2024-03-27T09:37:57.311 INFO:tasks.rados.rados.0.smithi170.stdout:667: done (3 left) 2024-03-27T09:37:57.311 INFO:tasks.rados.rados.0.smithi170.stdout:668: done (2 left) 2024-03-27T09:37:57.311 INFO:tasks.rados.rados.0.smithi170.stdout:669: done (1 left) 2024-03-27T09:37:57.311 INFO:tasks.rados.rados.0.smithi170.stdout:670: done (0 left) 2024-03-27T09:37:57.311 INFO:tasks.rados.rados.0.smithi170.stdout:671: read oid 2 snap 80 2024-03-27T09:37:57.312 INFO:tasks.rados.rados.0.smithi170.stdout:671: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:37:57.312 INFO:tasks.rados.rados.0.smithi170.stdout:672: delete oid 3 current snap is 81 2024-03-27T09:37:57.339 INFO:tasks.rados.rados.0.smithi170.stdout:672: done (1 left) 2024-03-27T09:37:57.339 INFO:tasks.rados.rados.0.smithi170.stdout:673: read oid 47 snap -1 2024-03-27T09:37:57.339 INFO:tasks.rados.rados.0.smithi170.stdout:673: expect (ObjNum 322 snap 81 seq_num 322) 2024-03-27T09:37:57.339 INFO:tasks.rados.rados.0.smithi170.stdout:674: delete oid 44 current snap is 81 2024-03-27T09:37:57.344 INFO:tasks.rados.rados.0.smithi170.stdout:671: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:37:57.458 INFO:tasks.rados.rados.0.smithi170.stdout:673: expect (ObjNum 322 snap 81 seq_num 322) 2024-03-27T09:37:57.675 INFO:tasks.rados.rados.0.smithi170.stdout:671: done (2 left) 2024-03-27T09:37:57.675 INFO:tasks.rados.rados.0.smithi170.stdout:673: done (1 left) 2024-03-27T09:37:57.675 INFO:tasks.rados.rados.0.smithi170.stdout:674: done (0 left) 2024-03-27T09:37:57.675 INFO:tasks.rados.rados.0.smithi170.stdout:675: snap_remove snap 51 2024-03-27T09:37:58.080 INFO:tasks.rados.rados.0.smithi170.stdout:675: done (0 left) 2024-03-27T09:37:58.081 INFO:tasks.rados.rados.0.smithi170.stdout:676: snap_create 2024-03-27T09:37:58.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:37:58.585 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:58.583+0000 7ffa9902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:58.588 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:37:58.588+0000 7ffa9902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:37:58.612 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:37:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:37:58.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:37:58.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:37:59.000 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:37:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:37:59.003 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:37:59.084 INFO:tasks.rados.rados.0.smithi170.stdout:676: done (0 left) 2024-03-27T09:37:59.085 INFO:tasks.rados.rados.0.smithi170.stdout:677: read oid 9 snap -1 2024-03-27T09:37:59.085 INFO:tasks.rados.rados.0.smithi170.stdout:677: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:59.085 INFO:tasks.rados.rados.0.smithi170.stdout:678: delete oid 16 current snap is 82 2024-03-27T09:37:59.131 INFO:tasks.rados.rados.0.smithi170.stdout:677: expect (ObjNum 257 snap 70 seq_num 257) 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:677: done (1 left) 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:678: done (0 left) 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:679: read oid 21 snap 80 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:679: expect deleted 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:680: read oid 14 snap 77 2024-03-27T09:37:59.288 INFO:tasks.rados.rados.0.smithi170.stdout:680: expect (ObjNum 232 snap 55 seq_num 232) 2024-03-27T09:37:59.289 INFO:tasks.rados.rados.0.smithi170.stdout:681: snap_remove snap 58 2024-03-27T09:38:00.088 INFO:tasks.rados.rados.0.smithi170.stdout:681: done (2 left) 2024-03-27T09:38:00.088 INFO:tasks.rados.rados.0.smithi170.stdout:682: snap_create 2024-03-27T09:38:00.088 INFO:tasks.rados.rados.0.smithi170.stdout:680: expect (ObjNum 232 snap 55 seq_num 232) 2024-03-27T09:38:00.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:00.767 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:00.765+0000 7f0984eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:00.771 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:00.770+0000 7f0984eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:00.794 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:00.841 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:01.089 INFO:tasks.rados.rados.0.smithi170.stdout:679: done (2 left) 2024-03-27T09:38:01.089 INFO:tasks.rados.rados.0.smithi170.stdout:680: done (1 left) 2024-03-27T09:38:01.089 INFO:tasks.rados.rados.0.smithi170.stdout:682: done (0 left) 2024-03-27T09:38:01.089 INFO:tasks.rados.rados.0.smithi170.stdout:683: read oid 46 snap -1 2024-03-27T09:38:01.090 INFO:tasks.rados.rados.0.smithi170.stdout:683: expect (ObjNum 277 snap 74 seq_num 277) 2024-03-27T09:38:01.090 INFO:tasks.rados.rados.0.smithi170.stdout:684: write (excl) oid 9 current snap is 83 2024-03-27T09:38:01.090 INFO:tasks.rados.rados.0.smithi170.stdout:684: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-03-27T09:38:01.097 INFO:tasks.rados.rados.0.smithi170.stdout:684: writing smithi17031214-9 from 746310 to 1381659 tid 1 2024-03-27T09:38:01.104 INFO:tasks.rados.rados.0.smithi170.stdout:684: writing smithi17031214-9 from 2077646 to 2654089 tid 2 2024-03-27T09:38:01.110 INFO:tasks.rados.rados.0.smithi170.stdout:684: writing smithi17031214-9 from 3362764 to 3546310 tid 3 2024-03-27T09:38:01.110 INFO:tasks.rados.rados.0.smithi170.stdout:685: write oid 28 current snap is 83 2024-03-27T09:38:01.111 INFO:tasks.rados.rados.0.smithi170.stdout:685: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-03-27T09:38:01.121 INFO:tasks.rados.rados.0.smithi170.stdout:685: writing smithi17031214-28 from 763117 to 1473715 tid 1 2024-03-27T09:38:01.133 INFO:tasks.rados.rados.0.smithi170.stdout:685: writing smithi17031214-28 from 2136128 to 2856229 tid 2 2024-03-27T09:38:01.137 INFO:tasks.rados.rados.0.smithi170.stdout:685: writing smithi17031214-28 from 3390187 to 3563117 tid 3 2024-03-27T09:38:01.138 INFO:tasks.rados.rados.0.smithi170.stdout:686: write (excl) oid 33 current snap is 83 2024-03-27T09:38:01.138 INFO:tasks.rados.rados.0.smithi170.stdout:686: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-03-27T09:38:01.149 INFO:tasks.rados.rados.0.smithi170.stdout:686: writing smithi17031214-33 from 779924 to 1565771 tid 1 2024-03-27T09:38:01.158 INFO:tasks.rados.rados.0.smithi170.stdout:686: writing smithi17031214-33 from 2078257 to 2658369 tid 2 2024-03-27T09:38:01.165 INFO:tasks.rados.rados.0.smithi170.stdout:686: writing smithi17031214-33 from 3301257 to 3579924 tid 3 2024-03-27T09:38:01.165 INFO:tasks.rados.rados.0.smithi170.stdout:687: delete oid 24 current snap is 83 2024-03-27T09:38:01.171 INFO:tasks.rados.rados.0.smithi170.stdout:683: expect (ObjNum 277 snap 74 seq_num 277) 2024-03-27T09:38:01.323 INFO:tasks.rados.rados.0.smithi170.stdout:686: finishing write tid 1 to smithi17031214-33 2024-03-27T09:38:01.323 INFO:tasks.rados.rados.0.smithi170.stdout:684: finishing write tid 1 to smithi17031214-9 2024-03-27T09:38:01.323 INFO:tasks.rados.rados.0.smithi170.stdout:685: finishing write tid 1 to smithi17031214-28 2024-03-27T09:38:01.333 INFO:tasks.rados.rados.0.smithi170.stdout:686: finishing write tid 2 to smithi17031214-33 2024-03-27T09:38:01.340 INFO:tasks.rados.rados.0.smithi170.stdout:684: finishing write tid 2 to smithi17031214-9 2024-03-27T09:38:01.343 INFO:tasks.rados.rados.0.smithi170.stdout:683: done (4 left) 2024-03-27T09:38:01.343 INFO:tasks.rados.rados.0.smithi170.stdout:687: done (3 left) 2024-03-27T09:38:01.343 INFO:tasks.rados.rados.0.smithi170.stdout:688: write (excl) oid 44 current snap is 83 2024-03-27T09:38:01.343 INFO:tasks.rados.rados.0.smithi170.stdout:688: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-03-27T09:38:01.349 INFO:tasks.rados.rados.0.smithi170.stdout:688: writing smithi17031214-44 from 413538 to 949883 tid 1 2024-03-27T09:38:01.357 INFO:tasks.rados.rados.0.smithi170.stdout:688: writing smithi17031214-44 from 1395221 to 1979002 tid 2 2024-03-27T09:38:01.366 INFO:tasks.rados.rados.0.smithi170.stdout:688: writing smithi17031214-44 from 2556103 to 3214720 tid 3 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:689: delete oid 23 current snap is 83 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:684: finishing write tid 3 to smithi17031214-9 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:685: finishing write tid 2 to smithi17031214-28 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:685: finishing write tid 3 to smithi17031214-28 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:686: finishing write tid 3 to smithi17031214-33 2024-03-27T09:38:01.367 INFO:tasks.rados.rados.0.smithi170.stdout:686: finishing write tid 4 to smithi17031214-33 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: finishing write tid 5 to smithi17031214-33 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: oid 33 updating version 0 to 179 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: oid 33 version 179 is already newer than 176 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: oid 33 version 179 is already newer than 179 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: oid 33 version 179 is already newer than 177 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 179 (ObjNum 332 snap 83 seq_num 332) dirty exists 2024-03-27T09:38:01.369 INFO:tasks.rados.rados.0.smithi170.stdout:686: left oid 33 (ObjNum 332 snap 83 seq_num 332) 2024-03-27T09:38:01.372 INFO:tasks.rados.rados.0.smithi170.stdout:684: finishing write tid 4 to smithi17031214-9 2024-03-27T09:38:01.372 INFO:tasks.rados.rados.0.smithi170.stdout:685: finishing write tid 4 to smithi17031214-28 2024-03-27T09:38:01.373 INFO:tasks.rados.rados.0.smithi170.stdout:688: finishing write tid 1 to smithi17031214-44 2024-03-27T09:38:01.374 INFO:tasks.rados.rados.0.smithi170.stdout:684: finishing write tid 5 to smithi17031214-9 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:684: oid 9 updating version 0 to 301 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:684: oid 9 version 301 is already newer than 299 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:684: oid 9 version 301 is already newer than 301 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:684: oid 9 updating version 301 to 302 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 302 (ObjNum 330 snap 83 seq_num 330) dirty exists 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:684: left oid 9 (ObjNum 330 snap 83 seq_num 330) 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:685: finishing write tid 5 to smithi17031214-28 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:685: oid 28 updating version 0 to 277 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:685: oid 28 version 277 is already newer than 275 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:685: oid 28 version 277 is already newer than 277 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:685: oid 28 updating version 277 to 278 2024-03-27T09:38:01.375 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 278 (ObjNum 331 snap 83 seq_num 331) dirty exists 2024-03-27T09:38:01.376 INFO:tasks.rados.rados.0.smithi170.stdout:685: left oid 28 (ObjNum 331 snap 83 seq_num 331) 2024-03-27T09:38:01.406 INFO:tasks.rados.rados.0.smithi170.stdout:688: finishing write tid 2 to smithi17031214-44 2024-03-27T09:38:01.417 INFO:tasks.rados.rados.0.smithi170.stdout:684: done (4 left) 2024-03-27T09:38:01.417 INFO:tasks.rados.rados.0.smithi170.stdout:685: done (3 left) 2024-03-27T09:38:01.417 INFO:tasks.rados.rados.0.smithi170.stdout:686: done (2 left) 2024-03-27T09:38:01.417 INFO:tasks.rados.rados.0.smithi170.stdout:689: done (1 left) 2024-03-27T09:38:01.417 INFO:tasks.rados.rados.0.smithi170.stdout:690: snap_remove snap 80 2024-03-27T09:38:02.096 INFO:tasks.rados.rados.0.smithi170.stdout:690: done (1 left) 2024-03-27T09:38:02.096 INFO:tasks.rados.rados.0.smithi170.stdout:691: write oid 29 current snap is 83 2024-03-27T09:38:02.096 INFO:tasks.rados.rados.0.smithi170.stdout:688: finishing write tid 3 to smithi17031214-44 2024-03-27T09:38:02.096 INFO:tasks.rados.rados.0.smithi170.stdout:691: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-03-27T09:38:02.100 INFO:tasks.rados.rados.0.smithi170.stdout:691: writing smithi17031214-29 from 447152 to 850348 tid 1 2024-03-27T09:38:02.107 INFO:tasks.rados.rados.0.smithi170.stdout:691: writing smithi17031214-29 from 1512185 to 2099635 tid 2 2024-03-27T09:38:02.115 INFO:tasks.rados.rados.0.smithi170.stdout:691: writing smithi17031214-29 from 2727302 to 3408463 tid 3 2024-03-27T09:38:02.115 INFO:tasks.rados.rados.0.smithi170.stdout:692: snap_remove snap 77 2024-03-27T09:38:02.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~132s 2024-03-27T09:38:02.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:02.965 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:02.962+0000 7f423595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:02.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:02.968+0000 7f423595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:02.993 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:03.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:692: done (2 left) 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:693: snap_create 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: finishing write tid 4 to smithi17031214-44 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: finishing write tid 5 to smithi17031214-44 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: oid 44 updating version 0 to 279 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: oid 44 version 279 is already newer than 279 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: oid 44 version 279 is already newer than 276 2024-03-27T09:38:03.098 INFO:tasks.rados.rados.0.smithi170.stdout:688: oid 44 version 279 is already newer than 277 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 279 (ObjNum 334 snap 83 seq_num 334) dirty exists 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:688: left oid 44 (ObjNum 334 snap 83 seq_num 334) 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: finishing write tid 1 to smithi17031214-29 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: finishing write tid 2 to smithi17031214-29 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: finishing write tid 3 to smithi17031214-29 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: finishing write tid 4 to smithi17031214-29 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: finishing write tid 5 to smithi17031214-29 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: oid 29 updating version 0 to 259 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: oid 29 updating version 259 to 261 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: oid 29 version 261 is already newer than 258 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: oid 29 version 261 is already newer than 261 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 261 (ObjNum 336 snap 83 seq_num 336) dirty exists 2024-03-27T09:38:03.099 INFO:tasks.rados.rados.0.smithi170.stdout:691: left oid 29 (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:04.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:04.101 INFO:tasks.rados.rados.0.smithi170.stdout:688: done (2 left) 2024-03-27T09:38:04.101 INFO:tasks.rados.rados.0.smithi170.stdout:691: done (1 left) 2024-03-27T09:38:04.101 INFO:tasks.rados.rados.0.smithi170.stdout:693: done (0 left) 2024-03-27T09:38:04.102 INFO:tasks.rados.rados.0.smithi170.stdout:694: read oid 19 snap -1 2024-03-27T09:38:04.102 INFO:tasks.rados.rados.0.smithi170.stdout:694: expect (ObjNum 317 snap 81 seq_num 317) 2024-03-27T09:38:04.102 INFO:tasks.rados.rados.0.smithi170.stdout:695: read oid 30 snap -1 2024-03-27T09:38:04.102 INFO:tasks.rados.rados.0.smithi170.stdout:695: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-27T09:38:04.102 INFO:tasks.rados.rados.0.smithi170.stdout:696: delete oid 42 current snap is 84 2024-03-27T09:38:04.109 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:04.113 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:04.168 INFO:tasks.rados.rados.0.smithi170.stdout:695: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:695: done (2 left) 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:696: done (1 left) 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:697: read oid 13 snap 81 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:697: expect deleted 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:698: read oid 4 snap -1 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:698: expect deleted 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:699: read oid 2 snap -1 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:699: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:700: read oid 9 snap -1 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:700: expect (ObjNum 330 snap 83 seq_num 330) 2024-03-27T09:38:04.275 INFO:tasks.rados.rados.0.smithi170.stdout:701: snap_remove snap 82 2024-03-27T09:38:05.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:05.104 INFO:tasks.rados.rados.0.smithi170.stdout:701: done (5 left) 2024-03-27T09:38:05.104 INFO:tasks.rados.rados.0.smithi170.stdout:702: read oid 29 snap 78 2024-03-27T09:38:05.104 INFO:tasks.rados.rados.0.smithi170.stdout:702: expect (ObjNum 228 snap 55 seq_num 228) 2024-03-27T09:38:05.105 INFO:tasks.rados.rados.0.smithi170.stdout:703: delete oid 20 current snap is 84 2024-03-27T09:38:05.105 INFO:tasks.rados.rados.0.smithi170.stdout:694: expect (ObjNum 317 snap 81 seq_num 317) 2024-03-27T09:38:05.155 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:05.153+0000 7f5c264ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:05.159 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:05.158+0000 7f5c264ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:05.185 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:05.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:05.234 INFO:tasks.rados.rados.0.smithi170.stdout:694: done (6 left) 2024-03-27T09:38:05.234 INFO:tasks.rados.rados.0.smithi170.stdout:703: done (5 left) 2024-03-27T09:38:05.234 INFO:tasks.rados.rados.0.smithi170.stdout:704: snap_remove snap 83 2024-03-27T09:38:05.965 INFO:tasks.rados.rados.0.smithi170.stdout:704: done (5 left) 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:705: read oid 36 snap 78 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:705: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:706: read oid 24 snap -1 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:706: expect deleted 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:707: read oid 40 snap 78 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:707: expect (ObjNum 281 snap 75 seq_num 281) 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:708: read oid 17 snap -1 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:708: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:709: delete oid 12 current snap is 84 2024-03-27T09:38:05.966 INFO:tasks.rados.rados.0.smithi170.stdout:699: expect (ObjNum 286 snap 78 seq_num 286) 2024-03-27T09:38:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:700: expect (ObjNum 330 snap 83 seq_num 330) 2024-03-27T09:38:06.304 INFO:tasks.rados.rados.0.smithi170.stdout:702: expect (ObjNum 228 snap 55 seq_num 228) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:697: done (9 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:698: done (8 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:699: done (7 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:700: done (6 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:702: done (5 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:709: done (4 left) 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:710: write oid 34 current snap is 84 2024-03-27T09:38:06.450 INFO:tasks.rados.rados.0.smithi170.stdout:710: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-03-27T09:38:06.454 INFO:tasks.rados.rados.0.smithi170.stdout:710: writing smithi17031214-34 from 514380 to 1218572 tid 1 2024-03-27T09:38:06.460 INFO:tasks.rados.rados.0.smithi170.stdout:710: writing smithi17031214-34 from 1629760 to 2108195 tid 2 2024-03-27T09:38:06.465 INFO:tasks.rados.rados.0.smithi170.stdout:710: writing smithi17031214-34 from 2604288 to 3046890 tid 3 2024-03-27T09:38:06.465 INFO:tasks.rados.rados.0.smithi170.stdout:706: done (4 left) 2024-03-27T09:38:06.465 INFO:tasks.rados.rados.0.smithi170.stdout:711: snap_remove snap 81 2024-03-27T09:38:07.111 INFO:tasks.rados.rados.0.smithi170.stdout:711: done (4 left) 2024-03-27T09:38:07.111 INFO:tasks.rados.rados.0.smithi170.stdout:712: delete oid 38 current snap is 84 2024-03-27T09:38:07.111 INFO:tasks.rados.rados.0.smithi170.stdout:707: expect (ObjNum 281 snap 75 seq_num 281) 2024-03-27T09:38:07.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:07.342 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:07.340+0000 7f5682f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:07.346 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:07.345+0000 7f5682f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:07.370 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:07.372 INFO:tasks.rados.rados.0.smithi170.stdout:705: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:07.414 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:07.630 INFO:tasks.rados.rados.0.smithi170.stdout:708: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: finishing write tid 1 to smithi17031214-34 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: finishing write tid 2 to smithi17031214-34 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: finishing write tid 3 to smithi17031214-34 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: finishing write tid 4 to smithi17031214-34 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: finishing write tid 5 to smithi17031214-34 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: oid 34 updating version 0 to 322 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: oid 34 updating version 322 to 325 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: oid 34 version 325 is already newer than 323 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:710: oid 34 version 325 is already newer than 325 2024-03-27T09:38:07.957 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 325 (ObjNum 340 snap 84 seq_num 340) dirty exists 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:710: left oid 34 (ObjNum 340 snap 84 seq_num 340) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:705: done (4 left) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:707: done (3 left) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:708: done (2 left) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:710: done (1 left) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:712: done (0 left) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:713: read oid 30 snap 78 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:713: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:714: read oid 44 snap -1 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:714: expect (ObjNum 334 snap 83 seq_num 334) 2024-03-27T09:38:07.958 INFO:tasks.rados.rados.0.smithi170.stdout:715: snap_create 2024-03-27T09:38:07.977 INFO:tasks.rados.rados.0.smithi170.stdout:713: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-27T09:38:08.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2024-03-27T09:38:08.084 INFO:tasks.rados.rados.0.smithi170.stdout:714: expect (ObjNum 334 snap 83 seq_num 334) 2024-03-27T09:38:08.223 INFO:tasks.rados.rados.0.smithi170.stdout:713: done (2 left) 2024-03-27T09:38:08.223 INFO:tasks.rados.rados.0.smithi170.stdout:714: done (1 left) 2024-03-27T09:38:08.223 INFO:tasks.rados.rados.0.smithi170.stdout:715: done (0 left) 2024-03-27T09:38:08.224 INFO:tasks.rados.rados.0.smithi170.stdout:716: delete oid 2 current snap is 85 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:716: done (0 left) 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:717: read oid 14 snap -1 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:717: expect deleted 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:718: read oid 50 snap 84 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:718: expect (ObjNum 285 snap 77 seq_num 285) 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:719: read oid 43 snap 84 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:719: expect deleted 2024-03-27T09:38:08.291 INFO:tasks.rados.rados.0.smithi170.stdout:720: snap_remove snap 78 2024-03-27T09:38:09.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:09.121 INFO:tasks.rados.rados.0.smithi170.stdout:720: done (3 left) 2024-03-27T09:38:09.121 INFO:tasks.rados.rados.0.smithi170.stdout:721: snap_create 2024-03-27T09:38:09.122 INFO:tasks.rados.rados.0.smithi170.stdout:718: expect (ObjNum 285 snap 77 seq_num 285) 2024-03-27T09:38:09.238 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:09.242 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:09.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:09.524 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:09.522+0000 7ffa68cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:09.528 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:09.527+0000 7ffa68cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:09.552 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:09.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:10.123 INFO:tasks.rados.rados.0.smithi170.stdout:717: done (3 left) 2024-03-27T09:38:10.123 INFO:tasks.rados.rados.0.smithi170.stdout:718: done (2 left) 2024-03-27T09:38:10.123 INFO:tasks.rados.rados.0.smithi170.stdout:719: done (1 left) 2024-03-27T09:38:10.123 INFO:tasks.rados.rados.0.smithi170.stdout:721: done (0 left) 2024-03-27T09:38:10.123 INFO:tasks.rados.rados.0.smithi170.stdout:722: snap_remove snap 84 2024-03-27T09:38:11.125 INFO:tasks.rados.rados.0.smithi170.stdout:722: done (0 left) 2024-03-27T09:38:11.125 INFO:tasks.rados.rados.0.smithi170.stdout:723: read oid 3 snap -1 2024-03-27T09:38:11.125 INFO:tasks.rados.rados.0.smithi170.stdout:723: expect deleted 2024-03-27T09:38:11.125 INFO:tasks.rados.rados.0.smithi170.stdout:724: write oid 12 current snap is 86 2024-03-27T09:38:11.125 INFO:tasks.rados.rados.0.smithi170.stdout:724: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-03-27T09:38:11.132 INFO:tasks.rados.rados.0.smithi170.stdout:724: writing smithi17031214-12 from 564801 to 1211093 tid 1 2024-03-27T09:38:11.141 INFO:tasks.rados.rados.0.smithi170.stdout:724: writing smithi17031214-12 from 1921559 to 2663674 tid 2 2024-03-27T09:38:11.148 INFO:tasks.rados.rados.0.smithi170.stdout:724: writing smithi17031214-12 from 3202910 to 3764801 tid 3 2024-03-27T09:38:11.148 INFO:tasks.rados.rados.0.smithi170.stdout:725: write oid 43 current snap is 86 2024-03-27T09:38:11.148 INFO:tasks.rados.rados.0.smithi170.stdout:725: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-03-27T09:38:11.157 INFO:tasks.rados.rados.0.smithi170.stdout:725: writing smithi17031214-43 from 581608 to 1303149 tid 1 2024-03-27T09:38:11.165 INFO:tasks.rados.rados.0.smithi170.stdout:725: writing smithi17031214-43 from 1863688 to 2349461 tid 2 2024-03-27T09:38:11.176 INFO:tasks.rados.rados.0.smithi170.stdout:725: writing smithi17031214-43 from 3113980 to 3718023 tid 3 2024-03-27T09:38:11.176 INFO:tasks.rados.rados.0.smithi170.stdout:726: snap_create 2024-03-27T09:38:11.256 INFO:tasks.rados.rados.0.smithi170.stdout:724: finishing write tid 1 to smithi17031214-12 2024-03-27T09:38:11.261 INFO:tasks.rados.rados.0.smithi170.stdout:725: finishing write tid 1 to smithi17031214-43 2024-03-27T09:38:11.282 INFO:tasks.rados.rados.0.smithi170.stdout:724: finishing write tid 2 to smithi17031214-12 2024-03-27T09:38:11.300 INFO:tasks.rados.rados.0.smithi170.stdout:725: finishing write tid 2 to smithi17031214-43 2024-03-27T09:38:11.317 INFO:tasks.rados.rados.0.smithi170.stdout:724: finishing write tid 3 to smithi17031214-12 2024-03-27T09:38:11.320 INFO:tasks.rados.rados.0.smithi170.stdout:724: finishing write tid 4 to smithi17031214-12 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:725: finishing write tid 3 to smithi17031214-43 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: finishing write tid 5 to smithi17031214-12 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: oid 12 updating version 0 to 281 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: oid 12 version 281 is already newer than 280 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: oid 12 updating version 281 to 283 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: oid 12 version 283 is already newer than 283 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 283 (ObjNum 343 snap 86 seq_num 343) dirty exists 2024-03-27T09:38:11.324 INFO:tasks.rados.rados.0.smithi170.stdout:724: left oid 12 (ObjNum 343 snap 86 seq_num 343) 2024-03-27T09:38:11.326 INFO:tasks.rados.rados.0.smithi170.stdout:725: finishing write tid 4 to smithi17031214-43 2024-03-27T09:38:11.328 INFO:tasks.rados.rados.0.smithi170.stdout:725: finishing write tid 5 to smithi17031214-43 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:725: oid 43 updating version 0 to 292 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:725: oid 43 version 292 is already newer than 291 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:725: oid 43 updating version 292 to 294 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:725: oid 43 version 294 is already newer than 294 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 294 (ObjNum 344 snap 86 seq_num 344) dirty exists 2024-03-27T09:38:11.329 INFO:tasks.rados.rados.0.smithi170.stdout:725: left oid 43 (ObjNum 344 snap 86 seq_num 344) 2024-03-27T09:38:11.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:11.713 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:11.711+0000 7fcfca95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:11.716 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:11.716+0000 7fcfca95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:11.740 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:11.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:12.131 INFO:tasks.rados.rados.0.smithi170.stdout:723: done (3 left) 2024-03-27T09:38:12.132 INFO:tasks.rados.rados.0.smithi170.stdout:724: done (2 left) 2024-03-27T09:38:12.132 INFO:tasks.rados.rados.0.smithi170.stdout:725: done (1 left) 2024-03-27T09:38:12.132 INFO:tasks.rados.rados.0.smithi170.stdout:726: done (0 left) 2024-03-27T09:38:12.132 INFO:tasks.rados.rados.0.smithi170.stdout:727: write (excl) oid 19 current snap is 87 2024-03-27T09:38:12.132 INFO:tasks.rados.rados.0.smithi170.stdout:727: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-03-27T09:38:12.140 INFO:tasks.rados.rados.0.smithi170.stdout:727: writing smithi17031214-19 from 598415 to 1395205 tid 1 2024-03-27T09:38:12.148 INFO:tasks.rados.rados.0.smithi170.stdout:727: writing smithi17031214-19 from 1922170 to 2667954 tid 2 2024-03-27T09:38:12.155 INFO:tasks.rados.rados.0.smithi170.stdout:727: writing smithi17031214-19 from 3141403 to 3556718 tid 3 2024-03-27T09:38:12.155 INFO:tasks.rados.rados.0.smithi170.stdout:728: write oid 9 current snap is 87 2024-03-27T09:38:12.155 INFO:tasks.rados.rados.0.smithi170.stdout:728: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-03-27T09:38:12.163 INFO:tasks.rados.rados.0.smithi170.stdout:728: writing smithi17031214-9 from 615222 to 1087261 tid 1 2024-03-27T09:38:12.170 INFO:tasks.rados.rados.0.smithi170.stdout:728: writing smithi17031214-9 from 1580652 to 2070094 tid 2 2024-03-27T09:38:12.180 INFO:tasks.rados.rados.0.smithi170.stdout:728: writing smithi17031214-9 from 2768826 to 3395413 tid 3 2024-03-27T09:38:12.180 INFO:tasks.rados.rados.0.smithi170.stdout:729: delete oid 5 current snap is 87 2024-03-27T09:38:12.229 INFO:tasks.rados.rados.0.smithi170.stdout:727: finishing write tid 1 to smithi17031214-19 2024-03-27T09:38:12.233 INFO:tasks.rados.rados.0.smithi170.stdout:728: finishing write tid 1 to smithi17031214-9 2024-03-27T09:38:12.283 INFO:tasks.rados.rados.0.smithi170.stdout:728: finishing write tid 2 to smithi17031214-9 2024-03-27T09:38:12.287 INFO:tasks.rados.rados.0.smithi170.stdout:727: finishing write tid 2 to smithi17031214-19 2024-03-27T09:38:12.287 INFO:tasks.rados.rados.0.smithi170.stdout:727: finishing write tid 3 to smithi17031214-19 2024-03-27T09:38:12.299 INFO:tasks.rados.rados.0.smithi170.stdout:729: done (2 left) 2024-03-27T09:38:12.300 INFO:tasks.rados.rados.0.smithi170.stdout:730: snap_remove snap 85 2024-03-27T09:38:13.131 INFO:tasks.rados.rados.0.smithi170.stdout:730: done (2 left) 2024-03-27T09:38:13.131 INFO:tasks.rados.rados.0.smithi170.stdout:731: read oid 6 snap -1 2024-03-27T09:38:13.131 INFO:tasks.rados.rados.0.smithi170.stdout:731: expect deleted 2024-03-27T09:38:13.131 INFO:tasks.rados.rados.0.smithi170.stdout:732: snap_remove snap 86 2024-03-27T09:38:13.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2024-03-27T09:38:13.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:13.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:13.897+0000 7f8ff495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:13.903 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:13.902+0000 7f8ff495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:13.926 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:14.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:14.137 INFO:tasks.rados.rados.0.smithi170.stdout:732: done (3 left) 2024-03-27T09:38:14.137 INFO:tasks.rados.rados.0.smithi170.stdout:733: delete oid 2 current snap is 87 2024-03-27T09:38:14.137 INFO:tasks.rados.rados.0.smithi170.stdout:728: finishing write tid 3 to smithi17031214-9 2024-03-27T09:38:14.137 INFO:tasks.rados.rados.0.smithi170.stdout:727: finishing write tid 4 to smithi17031214-19 2024-03-27T09:38:14.137 INFO:tasks.rados.rados.0.smithi170.stdout:727: finishing write tid 5 to smithi17031214-19 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:727: oid 19 updating version 0 to 220 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:727: oid 19 updating version 220 to 221 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:727: oid 19 version 221 is already newer than 220 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:727: oid 19 version 221 is already newer than 218 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 221 (ObjNum 345 snap 87 seq_num 345) dirty exists 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:727: left oid 19 (ObjNum 345 snap 87 seq_num 345) 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:728: finishing write tid 4 to smithi17031214-9 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:728: finishing write tid 5 to smithi17031214-9 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:728: oid 9 updating version 0 to 344 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:728: oid 9 version 344 is already newer than 343 2024-03-27T09:38:14.138 INFO:tasks.rados.rados.0.smithi170.stdout:728: oid 9 version 344 is already newer than 341 2024-03-27T09:38:14.139 INFO:tasks.rados.rados.0.smithi170.stdout:728: oid 9 version 344 is already newer than 343 2024-03-27T09:38:14.139 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 344 (ObjNum 346 snap 87 seq_num 346) dirty exists 2024-03-27T09:38:14.139 INFO:tasks.rados.rados.0.smithi170.stdout:728: left oid 9 (ObjNum 346 snap 87 seq_num 346) 2024-03-27T09:38:14.187 INFO:tasks.rados.rados.0.smithi170.stdout:727: done (3 left) 2024-03-27T09:38:14.187 INFO:tasks.rados.rados.0.smithi170.stdout:728: done (2 left) 2024-03-27T09:38:14.187 INFO:tasks.rados.rados.0.smithi170.stdout:731: done (1 left) 2024-03-27T09:38:14.188 INFO:tasks.rados.rados.0.smithi170.stdout:733: done (0 left) 2024-03-27T09:38:14.188 INFO:tasks.rados.rados.0.smithi170.stdout:734: snap_create 2024-03-27T09:38:14.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:14.342 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:14.346 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:15.140 INFO:tasks.rados.rados.0.smithi170.stdout:734: done (0 left) 2024-03-27T09:38:15.140 INFO:tasks.rados.rados.0.smithi170.stdout:735: write oid 12 current snap is 88 2024-03-27T09:38:15.141 INFO:tasks.rados.rados.0.smithi170.stdout:735: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-03-27T09:38:15.147 INFO:tasks.rados.rados.0.smithi170.stdout:735: writing smithi17031214-12 from 665643 to 1363429 tid 1 2024-03-27T09:38:15.155 INFO:tasks.rados.rados.0.smithi170.stdout:735: writing smithi17031214-12 from 2039745 to 2792867 tid 2 2024-03-27T09:38:15.161 INFO:tasks.rados.rados.0.smithi170.stdout:735: writing smithi17031214-12 from 3534742 to 3865643 tid 3 2024-03-27T09:38:15.161 INFO:tasks.rados.rados.0.smithi170.stdout:736: snap_remove snap 87 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:736: done (1 left) 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:737: delete oid 31 current snap is 88 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: finishing write tid 1 to smithi17031214-12 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: finishing write tid 2 to smithi17031214-12 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: finishing write tid 3 to smithi17031214-12 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: finishing write tid 4 to smithi17031214-12 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: finishing write tid 5 to smithi17031214-12 2024-03-27T09:38:15.967 INFO:tasks.rados.rados.0.smithi170.stdout:735: oid 12 updating version 0 to 286 2024-03-27T09:38:15.968 INFO:tasks.rados.rados.0.smithi170.stdout:735: oid 12 updating version 286 to 288 2024-03-27T09:38:15.968 INFO:tasks.rados.rados.0.smithi170.stdout:735: oid 12 version 288 is already newer than 288 2024-03-27T09:38:15.968 INFO:tasks.rados.rados.0.smithi170.stdout:735: oid 12 version 288 is already newer than 285 2024-03-27T09:38:15.968 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 288 (ObjNum 349 snap 88 seq_num 349) dirty exists 2024-03-27T09:38:15.968 INFO:tasks.rados.rados.0.smithi170.stdout:735: left oid 12 (ObjNum 349 snap 88 seq_num 349) 2024-03-27T09:38:16.037 INFO:tasks.rados.rados.0.smithi170.stdout:735: done (1 left) 2024-03-27T09:38:16.037 INFO:tasks.rados.rados.0.smithi170.stdout:737: done (0 left) 2024-03-27T09:38:16.038 INFO:tasks.rados.rados.0.smithi170.stdout:738: write (excl) oid 50 current snap is 88 2024-03-27T09:38:16.038 INFO:tasks.rados.rados.0.smithi170.stdout:738: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-03-27T09:38:16.044 INFO:tasks.rados.rados.0.smithi170.stdout:738: writing smithi17031214-50 from 699257 to 1263894 tid 1 2024-03-27T09:38:16.050 INFO:tasks.rados.rados.0.smithi170.stdout:738: writing smithi17031214-50 from 1873062 to 2513500 tid 2 2024-03-27T09:38:16.058 INFO:tasks.rados.rados.0.smithi170.stdout:738: writing smithi17031214-50 from 3305941 to 3899257 tid 3 2024-03-27T09:38:16.059 INFO:tasks.rados.rados.0.smithi170.stdout:739: snap_create 2024-03-27T09:38:16.070 INFO:tasks.rados.rados.0.smithi170.stdout:738: finishing write tid 1 to smithi17031214-50 2024-03-27T09:38:16.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:16.093 INFO:tasks.rados.rados.0.smithi170.stdout:738: finishing write tid 2 to smithi17031214-50 2024-03-27T09:38:16.098 INFO:tasks.rados.rados.0.smithi170.stdout:738: finishing write tid 3 to smithi17031214-50 2024-03-27T09:38:16.116 INFO:tasks.rados.rados.0.smithi170.stdout:738: finishing write tid 4 to smithi17031214-50 2024-03-27T09:38:16.117 INFO:tasks.rados.rados.0.smithi170.stdout:738: finishing write tid 5 to smithi17031214-50 2024-03-27T09:38:16.117 INFO:tasks.rados.rados.0.smithi170.stdout:738: oid 50 updating version 0 to 288 2024-03-27T09:38:16.117 INFO:tasks.rados.rados.0.smithi170.stdout:738: oid 50 version 288 is already newer than 286 2024-03-27T09:38:16.118 INFO:tasks.rados.rados.0.smithi170.stdout:738: oid 50 updating version 288 to 289 2024-03-27T09:38:16.118 INFO:tasks.rados.rados.0.smithi170.stdout:738: oid 50 version 289 is already newer than 288 2024-03-27T09:38:16.118 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 289 (ObjNum 351 snap 88 seq_num 351) dirty exists 2024-03-27T09:38:16.118 INFO:tasks.rados.rados.0.smithi170.stdout:738: left oid 50 (ObjNum 351 snap 88 seq_num 351) 2024-03-27T09:38:16.204 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:16.201+0000 7f01002ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:16.207 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:16.207+0000 7f01002ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:16.231 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:16.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:16.968 INFO:tasks.rados.rados.0.smithi170.stdout:738: done (1 left) 2024-03-27T09:38:16.968 INFO:tasks.rados.rados.0.smithi170.stdout:739: done (0 left) 2024-03-27T09:38:16.968 INFO:tasks.rados.rados.0.smithi170.stdout:740: read oid 17 snap -1 2024-03-27T09:38:16.968 INFO:tasks.rados.rados.0.smithi170.stdout:740: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:38:16.968 INFO:tasks.rados.rados.0.smithi170.stdout:741: snap_create 2024-03-27T09:38:17.081 INFO:tasks.rados.rados.0.smithi170.stdout:740: expect (ObjNum 220 snap 49 seq_num 220) 2024-03-27T09:38:17.968 INFO:tasks.rados.rados.0.smithi170.stdout:740: done (1 left) 2024-03-27T09:38:17.968 INFO:tasks.rados.rados.0.smithi170.stdout:741: done (0 left) 2024-03-27T09:38:17.968 INFO:tasks.rados.rados.0.smithi170.stdout:742: read oid 29 snap -1 2024-03-27T09:38:17.968 INFO:tasks.rados.rados.0.smithi170.stdout:742: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:17.969 INFO:tasks.rados.rados.0.smithi170.stdout:743: write oid 35 current snap is 90 2024-03-27T09:38:17.969 INFO:tasks.rados.rados.0.smithi170.stdout:743: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-03-27T09:38:17.975 INFO:tasks.rados.rados.0.smithi170.stdout:743: writing smithi17031214-35 from 716064 to 1355950 tid 1 2024-03-27T09:38:17.981 INFO:tasks.rados.rados.0.smithi170.stdout:743: writing smithi17031214-35 from 1815191 to 2315640 tid 2 2024-03-27T09:38:17.987 INFO:tasks.rados.rados.0.smithi170.stdout:743: writing smithi17031214-35 from 2817011 to 3343936 tid 3 2024-03-27T09:38:17.987 INFO:tasks.rados.rados.0.smithi170.stdout:744: read oid 46 snap -1 2024-03-27T09:38:17.987 INFO:tasks.rados.rados.0.smithi170.stdout:744: expect (ObjNum 277 snap 74 seq_num 277) 2024-03-27T09:38:17.988 INFO:tasks.rados.rados.0.smithi170.stdout:745: read oid 9 snap -1 2024-03-27T09:38:17.988 INFO:tasks.rados.rados.0.smithi170.stdout:745: expect (ObjNum 346 snap 87 seq_num 346) 2024-03-27T09:38:17.988 INFO:tasks.rados.rados.0.smithi170.stdout:746: delete oid 30 current snap is 90 2024-03-27T09:38:18.052 INFO:tasks.rados.rados.0.smithi170.stdout:743: finishing write tid 1 to smithi17031214-35 2024-03-27T09:38:18.054 INFO:tasks.rados.rados.0.smithi170.stdout:744: expect (ObjNum 277 snap 74 seq_num 277) 2024-03-27T09:38:18.164 INFO:tasks.rados.rados.0.smithi170.stdout:742: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:18.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:18.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:18.384+0000 7f54da82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:18.390 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:18.389+0000 7f54da82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:18.391 INFO:tasks.rados.rados.0.smithi170.stdout:743: finishing write tid 2 to smithi17031214-35 2024-03-27T09:38:18.391 INFO:tasks.rados.rados.0.smithi170.stdout:745: expect (ObjNum 346 snap 87 seq_num 346) 2024-03-27T09:38:18.414 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:18.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: finishing write tid 3 to smithi17031214-35 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: finishing write tid 4 to smithi17031214-35 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: finishing write tid 5 to smithi17031214-35 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: oid 35 updating version 0 to 305 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: oid 35 version 305 is already newer than 303 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: oid 35 version 305 is already newer than 302 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: oid 35 version 305 is already newer than 304 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 305 (ObjNum 352 snap 90 seq_num 352) dirty exists 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:743: left oid 35 (ObjNum 352 snap 90 seq_num 352) 2024-03-27T09:38:18.661 INFO:tasks.rados.rados.0.smithi170.stdout:742: done (4 left) 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:743: done (3 left) 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:744: done (2 left) 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:745: done (1 left) 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:746: done (0 left) 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:747: write oid 15 current snap is 90 2024-03-27T09:38:18.662 INFO:tasks.rados.rados.0.smithi170.stdout:747: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-03-27T09:38:18.668 INFO:tasks.rados.rados.0.smithi170.stdout:747: writing smithi17031214-15 from 749678 to 1540062 tid 1 2024-03-27T09:38:18.675 INFO:tasks.rados.rados.0.smithi170.stdout:747: writing smithi17031214-15 from 2332155 to 2836273 tid 2 2024-03-27T09:38:18.681 INFO:tasks.rados.rados.0.smithi170.stdout:747: writing smithi17031214-15 from 3271857 to 3821326 tid 3 2024-03-27T09:38:18.681 INFO:tasks.rados.rados.0.smithi170.stdout:748: read oid 5 snap 89 2024-03-27T09:38:18.681 INFO:tasks.rados.rados.0.smithi170.stdout:748: expect deleted 2024-03-27T09:38:18.682 INFO:tasks.rados.rados.0.smithi170.stdout:749: write oid 25 current snap is 90 2024-03-27T09:38:18.682 INFO:tasks.rados.rados.0.smithi170.stdout:749: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-03-27T09:38:18.688 INFO:tasks.rados.rados.0.smithi170.stdout:749: writing smithi17031214-25 from 766485 to 1232118 tid 1 2024-03-27T09:38:18.695 INFO:tasks.rados.rados.0.smithi170.stdout:749: writing smithi17031214-25 from 1990637 to 2638413 tid 2 2024-03-27T09:38:18.703 INFO:tasks.rados.rados.0.smithi170.stdout:749: writing smithi17031214-25 from 3299280 to 3966485 tid 3 2024-03-27T09:38:18.703 INFO:tasks.rados.rados.0.smithi170.stdout:750: write (excl) oid 44 current snap is 90 2024-03-27T09:38:18.703 INFO:tasks.rados.rados.0.smithi170.stdout:750: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-03-27T09:38:18.710 INFO:tasks.rados.rados.0.smithi170.stdout:750: writing smithi17031214-44 from 783292 to 1324174 tid 1 2024-03-27T09:38:18.716 INFO:tasks.rados.rados.0.smithi170.stdout:750: writing smithi17031214-44 from 1932766 to 2440553 tid 2 2024-03-27T09:38:18.725 INFO:tasks.rados.rados.0.smithi170.stdout:750: writing smithi17031214-44 from 3210350 to 3782363 tid 3 2024-03-27T09:38:18.725 INFO:tasks.rados.rados.0.smithi170.stdout:751: delete oid 22 current snap is 90 2024-03-27T09:38:18.725 INFO:tasks.rados.rados.0.smithi170.stdout:749: finishing write tid 1 to smithi17031214-25 2024-03-27T09:38:18.779 INFO:tasks.rados.rados.0.smithi170.stdout:749: finishing write tid 2 to smithi17031214-25 2024-03-27T09:38:18.799 INFO:tasks.rados.rados.0.smithi170.stdout:750: finishing write tid 1 to smithi17031214-44 2024-03-27T09:38:18.801 INFO:tasks.rados.rados.0.smithi170.stdout:747: finishing write tid 1 to smithi17031214-15 2024-03-27T09:38:18.811 INFO:tasks.rados.rados.0.smithi170.stdout:748: done (4 left) 2024-03-27T09:38:18.811 INFO:tasks.rados.rados.0.smithi170.stdout:751: done (3 left) 2024-03-27T09:38:18.811 INFO:tasks.rados.rados.0.smithi170.stdout:752: delete oid 13 current snap is 90 2024-03-27T09:38:18.820 INFO:tasks.rados.rados.0.smithi170.stdout:750: finishing write tid 2 to smithi17031214-44 2024-03-27T09:38:18.822 INFO:tasks.rados.rados.0.smithi170.stdout:752: done (3 left) 2024-03-27T09:38:18.822 INFO:tasks.rados.rados.0.smithi170.stdout:753: snap_create 2024-03-27T09:38:18.828 INFO:tasks.rados.rados.0.smithi170.stdout:749: finishing write tid 3 to smithi17031214-25 2024-03-27T09:38:18.839 INFO:tasks.rados.rados.0.smithi170.stdout:747: finishing write tid 2 to smithi17031214-15 2024-03-27T09:38:18.844 INFO:tasks.rados.rados.0.smithi170.stdout:749: finishing write tid 4 to smithi17031214-25 2024-03-27T09:38:18.848 INFO:tasks.rados.rados.0.smithi170.stdout:749: finishing write tid 5 to smithi17031214-25 2024-03-27T09:38:18.848 INFO:tasks.rados.rados.0.smithi170.stdout:749: oid 25 updating version 0 to 299 2024-03-27T09:38:18.848 INFO:tasks.rados.rados.0.smithi170.stdout:749: oid 25 version 299 is already newer than 298 2024-03-27T09:38:18.848 INFO:tasks.rados.rados.0.smithi170.stdout:749: oid 25 version 299 is already newer than 297 2024-03-27T09:38:18.849 INFO:tasks.rados.rados.0.smithi170.stdout:749: oid 25 version 299 is already newer than 296 2024-03-27T09:38:18.849 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 299 (ObjNum 355 snap 90 seq_num 355) dirty exists 2024-03-27T09:38:18.849 INFO:tasks.rados.rados.0.smithi170.stdout:749: left oid 25 (ObjNum 355 snap 90 seq_num 355) 2024-03-27T09:38:18.850 INFO:tasks.rados.rados.0.smithi170.stdout:750: finishing write tid 3 to smithi17031214-44 2024-03-27T09:38:18.862 INFO:tasks.rados.rados.0.smithi170.stdout:750: finishing write tid 4 to smithi17031214-44 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:750: finishing write tid 5 to smithi17031214-44 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:750: oid 44 updating version 0 to 291 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:750: oid 44 updating version 291 to 292 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:750: oid 44 updating version 292 to 294 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:750: oid 44 version 294 is already newer than 293 2024-03-27T09:38:18.863 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 294 (ObjNum 356 snap 90 seq_num 356) dirty exists 2024-03-27T09:38:18.864 INFO:tasks.rados.rados.0.smithi170.stdout:750: left oid 44 (ObjNum 356 snap 90 seq_num 356) 2024-03-27T09:38:18.864 INFO:tasks.rados.rados.0.smithi170.stdout:747: finishing write tid 3 to smithi17031214-15 2024-03-27T09:38:18.872 INFO:tasks.rados.rados.0.smithi170.stdout:747: finishing write tid 4 to smithi17031214-15 2024-03-27T09:38:18.873 INFO:tasks.rados.rados.0.smithi170.stdout:747: finishing write tid 5 to smithi17031214-15 2024-03-27T09:38:18.873 INFO:tasks.rados.rados.0.smithi170.stdout:747: oid 15 updating version 0 to 124 2024-03-27T09:38:18.873 INFO:tasks.rados.rados.0.smithi170.stdout:747: oid 15 version 124 is already newer than 123 2024-03-27T09:38:18.873 INFO:tasks.rados.rados.0.smithi170.stdout:747: oid 15 version 124 is already newer than 122 2024-03-27T09:38:18.874 INFO:tasks.rados.rados.0.smithi170.stdout:747: oid 15 updating version 124 to 125 2024-03-27T09:38:18.874 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 125 (ObjNum 354 snap 90 seq_num 354) dirty exists 2024-03-27T09:38:18.874 INFO:tasks.rados.rados.0.smithi170.stdout:747: left oid 15 (ObjNum 354 snap 90 seq_num 354) 2024-03-27T09:38:19.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~149s 2024-03-27T09:38:19.145 INFO:tasks.rados.rados.0.smithi170.stdout:747: done (3 left) 2024-03-27T09:38:19.145 INFO:tasks.rados.rados.0.smithi170.stdout:749: done (2 left) 2024-03-27T09:38:19.145 INFO:tasks.rados.rados.0.smithi170.stdout:750: done (1 left) 2024-03-27T09:38:19.145 INFO:tasks.rados.rados.0.smithi170.stdout:753: done (0 left) 2024-03-27T09:38:19.145 INFO:tasks.rados.rados.0.smithi170.stdout:754: snap_remove snap 88 2024-03-27T09:38:19.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:19.446 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:19.450 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:20.148 INFO:tasks.rados.rados.0.smithi170.stdout:754: done (0 left) 2024-03-27T09:38:20.148 INFO:tasks.rados.rados.0.smithi170.stdout:755: snap_remove snap 90 2024-03-27T09:38:20.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:20.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:20.572+0000 7f523a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:20.579 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:20.578+0000 7f523a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:20.603 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:20.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:21.151 INFO:tasks.rados.rados.0.smithi170.stdout:755: done (0 left) 2024-03-27T09:38:21.151 INFO:tasks.rados.rados.0.smithi170.stdout:756: write oid 26 current snap is 91 2024-03-27T09:38:21.151 INFO:tasks.rados.rados.0.smithi170.stdout:756: seq_num 359 ranges {433713=482982,1424565=609148} 2024-03-27T09:38:21.156 INFO:tasks.rados.rados.0.smithi170.stdout:756: writing smithi17031214-26 from 433713 to 916695 tid 1 2024-03-27T09:38:21.162 INFO:tasks.rados.rados.0.smithi170.stdout:756: writing smithi17031214-26 from 1424565 to 2033713 tid 2 2024-03-27T09:38:21.162 INFO:tasks.rados.rados.0.smithi170.stdout:757: read oid 29 snap -1 2024-03-27T09:38:21.163 INFO:tasks.rados.rados.0.smithi170.stdout:757: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:21.163 INFO:tasks.rados.rados.0.smithi170.stdout:758: write (excl) oid 9 current snap is 91 2024-03-27T09:38:21.163 INFO:tasks.rados.rados.0.smithi170.stdout:758: seq_num 360 ranges {450520=558231,1483047=515125} 2024-03-27T09:38:21.168 INFO:tasks.rados.rados.0.smithi170.stdout:758: writing smithi17031214-9 from 450520 to 1008751 tid 1 2024-03-27T09:38:21.174 INFO:tasks.rados.rados.0.smithi170.stdout:758: writing smithi17031214-9 from 1483047 to 1998172 tid 2 2024-03-27T09:38:21.174 INFO:tasks.rados.rados.0.smithi170.stdout:759: snap_remove snap 89 2024-03-27T09:38:22.155 INFO:tasks.rados.rados.0.smithi170.stdout:759: done (3 left) 2024-03-27T09:38:22.155 INFO:tasks.rados.rados.0.smithi170.stdout:760: delete oid 5 current snap is 91 2024-03-27T09:38:22.155 INFO:tasks.rados.rados.0.smithi170.stdout:756: finishing write tid 1 to smithi17031214-26 2024-03-27T09:38:22.155 INFO:tasks.rados.rados.0.smithi170.stdout:757: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:758: finishing write tid 1 to smithi17031214-9 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: finishing write tid 2 to smithi17031214-26 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: finishing write tid 3 to smithi17031214-26 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: finishing write tid 4 to smithi17031214-26 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: oid 26 updating version 0 to 229 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: oid 26 version 229 is already newer than 229 2024-03-27T09:38:22.383 INFO:tasks.rados.rados.0.smithi170.stdout:756: oid 26 version 229 is already newer than 227 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 229 (ObjNum 359 snap 91 seq_num 359) dirty exists 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:756: left oid 26 (ObjNum 359 snap 91 seq_num 359) 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: finishing write tid 2 to smithi17031214-9 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: finishing write tid 3 to smithi17031214-9 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:756: done (3 left) 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:757: done (2 left) 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:760: done (1 left) 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:761: write (excl) oid 40 current snap is 91 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: finishing write tid 4 to smithi17031214-9 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: oid 9 updating version 0 to 351 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: oid 9 version 351 is already newer than 350 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:758: oid 9 updating version 351 to 352 2024-03-27T09:38:22.384 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 352 (ObjNum 360 snap 91 seq_num 360) dirty exists 2024-03-27T09:38:22.385 INFO:tasks.rados.rados.0.smithi170.stdout:758: left oid 9 (ObjNum 360 snap 91 seq_num 360) 2024-03-27T09:38:22.385 INFO:tasks.rados.rados.0.smithi170.stdout:761: seq_num 362 ranges {484134=708729,1883658=200476} 2024-03-27T09:38:22.389 INFO:tasks.rados.rados.0.smithi170.stdout:761: writing smithi17031214-40 from 484134 to 1192863 tid 1 2024-03-27T09:38:22.394 INFO:tasks.rados.rados.0.smithi170.stdout:761: writing smithi17031214-40 from 1883658 to 2084134 tid 2 2024-03-27T09:38:22.394 INFO:tasks.rados.rados.0.smithi170.stdout:758: done (1 left) 2024-03-27T09:38:22.394 INFO:tasks.rados.rados.0.smithi170.stdout:762: write oid 30 current snap is 91 2024-03-27T09:38:22.394 INFO:tasks.rados.rados.0.smithi170.stdout:762: seq_num 363 ranges {500941=783978,1942140=158801} 2024-03-27T09:38:22.402 INFO:tasks.rados.rados.0.smithi170.stdout:762: writing smithi17031214-30 from 500941 to 1284919 tid 1 2024-03-27T09:38:22.407 INFO:tasks.rados.rados.0.smithi170.stdout:762: writing smithi17031214-30 from 1942140 to 2100941 tid 2 2024-03-27T09:38:22.407 INFO:tasks.rados.rados.0.smithi170.stdout:763: write (excl) oid 3 current snap is 91 2024-03-27T09:38:22.407 INFO:tasks.rados.rados.0.smithi170.stdout:763: seq_num 364 ranges {517748=459227,1600622=406110} 2024-03-27T09:38:22.414 INFO:tasks.rados.rados.0.smithi170.stdout:763: writing smithi17031214-3 from 517748 to 976975 tid 1 2024-03-27T09:38:22.422 INFO:tasks.rados.rados.0.smithi170.stdout:763: writing smithi17031214-3 from 1600622 to 2006732 tid 2 2024-03-27T09:38:22.422 INFO:tasks.rados.rados.0.smithi170.stdout:764: write (excl) oid 34 current snap is 91 2024-03-27T09:38:22.422 INFO:tasks.rados.rados.0.smithi170.stdout:764: seq_num 365 ranges {534555=650829,1659104=475451} 2024-03-27T09:38:22.432 INFO:tasks.rados.rados.0.smithi170.stdout:764: writing smithi17031214-34 from 534555 to 1185384 tid 1 2024-03-27T09:38:22.440 INFO:tasks.rados.rados.0.smithi170.stdout:764: writing smithi17031214-34 from 1659104 to 2134555 tid 2 2024-03-27T09:38:22.440 INFO:tasks.rados.rados.0.smithi170.stdout:765: write (excl) oid 10 current snap is 91 2024-03-27T09:38:22.440 INFO:tasks.rados.rados.0.smithi170.stdout:765: seq_num 366 ranges {551362=726078,1717586=409779} 2024-03-27T09:38:22.449 INFO:tasks.rados.rados.0.smithi170.stdout:765: writing smithi17031214-10 from 551362 to 1277440 tid 1 2024-03-27T09:38:22.456 INFO:tasks.rados.rados.0.smithi170.stdout:765: writing smithi17031214-10 from 1717586 to 2127365 tid 2 2024-03-27T09:38:22.456 INFO:tasks.rados.rados.0.smithi170.stdout:766: write (excl) oid 15 current snap is 91 2024-03-27T09:38:22.456 INFO:tasks.rados.rados.0.smithi170.stdout:766: seq_num 367 ranges {568169=401327,1376068=669790} 2024-03-27T09:38:22.463 INFO:tasks.rados.rados.0.smithi170.stdout:766: writing smithi17031214-15 from 568169 to 969496 tid 1 2024-03-27T09:38:22.471 INFO:tasks.rados.rados.0.smithi170.stdout:766: writing smithi17031214-15 from 1376068 to 2045858 tid 2 2024-03-27T09:38:22.471 INFO:tasks.rados.rados.0.smithi170.stdout:767: write (excl) oid 7 current snap is 91 2024-03-27T09:38:22.471 INFO:tasks.rados.rados.0.smithi170.stdout:767: seq_num 368 ranges {584976=476576,1834550=350426} 2024-03-27T09:38:22.477 INFO:tasks.rados.rados.0.smithi170.stdout:767: writing smithi17031214-7 from 584976 to 1061552 tid 1 2024-03-27T09:38:22.484 INFO:tasks.rados.rados.0.smithi170.stdout:767: writing smithi17031214-7 from 1834550 to 2184976 tid 2 2024-03-27T09:38:22.484 INFO:tasks.rados.rados.0.smithi170.stdout:768: read oid 49 snap -1 2024-03-27T09:38:22.484 INFO:tasks.rados.rados.0.smithi170.stdout:768: expect deleted 2024-03-27T09:38:22.484 INFO:tasks.rados.rados.0.smithi170.stdout:769: write oid 26 current snap is 91 2024-03-27T09:38:22.484 INFO:tasks.rados.rados.0.smithi170.stdout:769: seq_num 369 ranges {601783=551825,1776679=425104} 2024-03-27T09:38:22.493 INFO:tasks.rados.rados.0.smithi170.stdout:769: writing smithi17031214-26 from 601783 to 1153608 tid 1 2024-03-27T09:38:22.500 INFO:tasks.rados.rados.0.smithi170.stdout:769: writing smithi17031214-26 from 1776679 to 2201783 tid 2 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:761: finishing write tid 1 to smithi17031214-40 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:770: read oid 2 snap -1 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:770: expect deleted 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:771: read oid 24 snap -1 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:771: expect deleted 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:772: read oid 29 snap -1 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:772: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:773: read oid 13 snap -1 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:773: expect deleted 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:774: write oid 37 current snap is 91 2024-03-27T09:38:22.501 INFO:tasks.rados.rados.0.smithi170.stdout:774: seq_num 370 ranges {618590=627074,1835161=383429} 2024-03-27T09:38:22.509 INFO:tasks.rados.rados.0.smithi170.stdout:774: writing smithi17031214-37 from 618590 to 1245664 tid 1 2024-03-27T09:38:22.517 INFO:tasks.rados.rados.0.smithi170.stdout:774: writing smithi17031214-37 from 1835161 to 2218590 tid 2 2024-03-27T09:38:22.517 INFO:tasks.rados.rados.0.smithi170.stdout:775: write oid 18 current snap is 91 2024-03-27T09:38:22.517 INFO:tasks.rados.rados.0.smithi170.stdout:775: seq_num 371 ranges {635397=702323,1893643=341754} 2024-03-27T09:38:22.527 INFO:tasks.rados.rados.0.smithi170.stdout:775: writing smithi17031214-18 from 635397 to 1337720 tid 1 2024-03-27T09:38:22.534 INFO:tasks.rados.rados.0.smithi170.stdout:775: writing smithi17031214-18 from 1893643 to 2235397 tid 2 2024-03-27T09:38:22.534 INFO:tasks.rados.rados.0.smithi170.stdout:776: write (excl) oid 22 current snap is 91 2024-03-27T09:38:22.534 INFO:tasks.rados.rados.0.smithi170.stdout:776: seq_num 372 ranges {652204=777572,1952125=300079} 2024-03-27T09:38:22.545 INFO:tasks.rados.rados.0.smithi170.stdout:776: writing smithi17031214-22 from 652204 to 1429776 tid 1 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout:776: writing smithi17031214-22 from 1952125 to 2252204 tid 2 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout:763: finishing write tid 1 to smithi17031214-3 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout:763: finishing write tid 2 to smithi17031214-3 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout:763: finishing write tid 3 to smithi17031214-3 2024-03-27T09:38:22.552 INFO:tasks.rados.rados.0.smithi170.stdout:766: finishing write tid 1 to smithi17031214-15 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:762: finishing write tid 1 to smithi17031214-30 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: finishing write tid 4 to smithi17031214-3 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: oid 3 updating version 0 to 203 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: oid 3 version 203 is already newer than 202 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: oid 3 updating version 203 to 204 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 204 (ObjNum 364 snap 91 seq_num 364) dirty exists 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: left oid 3 (ObjNum 364 snap 91 seq_num 364) 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:763: done (15 left) 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:768: done (14 left) 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:777: write (excl) oid 31 current snap is 91 2024-03-27T09:38:22.553 INFO:tasks.rados.rados.0.smithi170.stdout:777: seq_num 373 ranges {669011=569174,2010607=258404} 2024-03-27T09:38:22.560 INFO:tasks.rados.rados.0.smithi170.stdout:777: writing smithi17031214-31 from 669011 to 1238185 tid 1 2024-03-27T09:38:22.567 INFO:tasks.rados.rados.0.smithi170.stdout:777: writing smithi17031214-31 from 2010607 to 2269011 tid 2 2024-03-27T09:38:22.567 INFO:tasks.rados.rados.0.smithi170.stdout:778: write oid 39 current snap is 91 2024-03-27T09:38:22.567 INFO:tasks.rados.rados.0.smithi170.stdout:774: finishing write tid 1 to smithi17031214-37 2024-03-27T09:38:22.567 INFO:tasks.rados.rados.0.smithi170.stdout:778: seq_num 374 ranges {685818=644423,2069089=216729} 2024-03-27T09:38:22.574 INFO:tasks.rados.rados.0.smithi170.stdout:778: writing smithi17031214-39 from 685818 to 1330241 tid 1 2024-03-27T09:38:22.580 INFO:tasks.rados.rados.0.smithi170.stdout:778: writing smithi17031214-39 from 2069089 to 2285818 tid 2 2024-03-27T09:38:22.580 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2024-03-27T09:38:22.580 INFO:tasks.rados.rados.0.smithi170.stdout:761: finishing write tid 2 to smithi17031214-40 2024-03-27T09:38:22.584 INFO:tasks.rados.rados.0.smithi170.stdout:774: finishing write tid 2 to smithi17031214-37 2024-03-27T09:38:22.617 INFO:tasks.rados.rados.0.smithi170.stdout:762: finishing write tid 2 to smithi17031214-30 2024-03-27T09:38:22.620 INFO:tasks.rados.rados.0.smithi170.stdout:774: finishing write tid 3 to smithi17031214-37 2024-03-27T09:38:22.625 INFO:tasks.rados.rados.0.smithi170.stdout:774: finishing write tid 4 to smithi17031214-37 2024-03-27T09:38:22.625 INFO:tasks.rados.rados.0.smithi170.stdout:774: oid 37 updating version 0 to 310 2024-03-27T09:38:22.625 INFO:tasks.rados.rados.0.smithi170.stdout:774: oid 37 updating version 310 to 311 2024-03-27T09:38:22.625 INFO:tasks.rados.rados.0.smithi170.stdout:774: oid 37 updating version 311 to 312 2024-03-27T09:38:22.626 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 312 (ObjNum 370 snap 91 seq_num 370) dirty exists 2024-03-27T09:38:22.626 INFO:tasks.rados.rados.0.smithi170.stdout:774: left oid 37 (ObjNum 370 snap 91 seq_num 370) 2024-03-27T09:38:22.626 INFO:tasks.rados.rados.0.smithi170.stdout:774: done (15 left) 2024-03-27T09:38:22.626 INFO:tasks.rados.rados.0.smithi170.stdout:779: delete oid 35 current snap is 91 2024-03-27T09:38:22.630 INFO:tasks.rados.rados.0.smithi170.stdout:762: finishing write tid 3 to smithi17031214-30 2024-03-27T09:38:22.635 INFO:tasks.rados.rados.0.smithi170.stdout:762: finishing write tid 4 to smithi17031214-30 2024-03-27T09:38:22.635 INFO:tasks.rados.rados.0.smithi170.stdout:762: oid 30 updating version 0 to 186 2024-03-27T09:38:22.636 INFO:tasks.rados.rados.0.smithi170.stdout:762: oid 30 updating version 186 to 188 2024-03-27T09:38:22.636 INFO:tasks.rados.rados.0.smithi170.stdout:762: oid 30 version 188 is already newer than 188 2024-03-27T09:38:22.636 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 188 (ObjNum 363 snap 91 seq_num 363) dirty exists 2024-03-27T09:38:22.636 INFO:tasks.rados.rados.0.smithi170.stdout:762: left oid 30 (ObjNum 363 snap 91 seq_num 363) 2024-03-27T09:38:22.641 INFO:tasks.rados.rados.0.smithi170.stdout:766: finishing write tid 2 to smithi17031214-15 2024-03-27T09:38:22.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:22.653 INFO:tasks.rados.rados.0.smithi170.stdout:775: finishing write tid 1 to smithi17031214-18 2024-03-27T09:38:22.656 INFO:tasks.rados.rados.0.smithi170.stdout:776: finishing write tid 1 to smithi17031214-22 2024-03-27T09:38:22.669 INFO:tasks.rados.rados.0.smithi170.stdout:778: finishing write tid 1 to smithi17031214-39 2024-03-27T09:38:22.669 INFO:tasks.rados.rados.0.smithi170.stdout:775: finishing write tid 2 to smithi17031214-18 2024-03-27T09:38:22.672 INFO:tasks.rados.rados.0.smithi170.stdout:761: finishing write tid 3 to smithi17031214-40 2024-03-27T09:38:22.673 INFO:tasks.rados.rados.0.smithi170.stdout:767: finishing write tid 1 to smithi17031214-7 2024-03-27T09:38:22.676 INFO:tasks.rados.rados.0.smithi170.stdout:761: finishing write tid 4 to smithi17031214-40 2024-03-27T09:38:22.676 INFO:tasks.rados.rados.0.smithi170.stdout:761: oid 40 updating version 0 to 357 2024-03-27T09:38:22.677 INFO:tasks.rados.rados.0.smithi170.stdout:761: oid 40 version 357 is already newer than 355 2024-03-27T09:38:22.677 INFO:tasks.rados.rados.0.smithi170.stdout:761: oid 40 version 357 is already newer than 356 2024-03-27T09:38:22.677 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 357 (ObjNum 362 snap 91 seq_num 362) dirty exists 2024-03-27T09:38:22.677 INFO:tasks.rados.rados.0.smithi170.stdout:761: left oid 40 (ObjNum 362 snap 91 seq_num 362) 2024-03-27T09:38:22.682 INFO:tasks.rados.rados.0.smithi170.stdout:765: finishing write tid 1 to smithi17031214-10 2024-03-27T09:38:22.687 INFO:tasks.rados.rados.0.smithi170.stdout:775: finishing write tid 3 to smithi17031214-18 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:775: finishing write tid 4 to smithi17031214-18 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:775: oid 18 updating version 0 to 315 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:775: oid 18 version 315 is already newer than 315 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:775: oid 18 version 315 is already newer than 315 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 315 (ObjNum 371 snap 91 seq_num 371) dirty exists 2024-03-27T09:38:22.693 INFO:tasks.rados.rados.0.smithi170.stdout:775: left oid 18 (ObjNum 371 snap 91 seq_num 371) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:761: done (15 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:762: done (14 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:770: done (13 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:773: done (12 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:775: done (11 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:779: done (10 left) 2024-03-27T09:38:22.726 INFO:tasks.rados.rados.0.smithi170.stdout:780: delete oid 9 current snap is 91 2024-03-27T09:38:22.735 INFO:tasks.rados.rados.0.smithi170.stdout:769: finishing write tid 1 to smithi17031214-26 2024-03-27T09:38:22.740 INFO:tasks.rados.rados.0.smithi170.stdout:766: finishing write tid 3 to smithi17031214-15 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:766: finishing write tid 4 to smithi17031214-15 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:766: oid 15 updating version 0 to 132 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:766: oid 15 version 132 is already newer than 131 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:766: oid 15 version 132 is already newer than 130 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 132 (ObjNum 367 snap 91 seq_num 367) dirty exists 2024-03-27T09:38:22.743 INFO:tasks.rados.rados.0.smithi170.stdout:766: left oid 15 (ObjNum 367 snap 91 seq_num 367) 2024-03-27T09:38:22.748 INFO:tasks.rados.rados.0.smithi170.stdout:776: finishing write tid 2 to smithi17031214-22 2024-03-27T09:38:22.760 INFO:tasks.rados.rados.0.smithi170.stdout:776: finishing write tid 3 to smithi17031214-22 2024-03-27T09:38:22.762 INFO:tasks.rados.rados.0.smithi170.stdout:776: finishing write tid 4 to smithi17031214-22 2024-03-27T09:38:22.762 INFO:tasks.rados.rados.0.smithi170.stdout:776: oid 22 updating version 0 to 174 2024-03-27T09:38:22.763 INFO:tasks.rados.rados.0.smithi170.stdout:776: oid 22 version 174 is already newer than 172 2024-03-27T09:38:22.763 INFO:tasks.rados.rados.0.smithi170.stdout:776: oid 22 version 174 is already newer than 174 2024-03-27T09:38:22.763 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 174 (ObjNum 372 snap 91 seq_num 372) dirty exists 2024-03-27T09:38:22.763 INFO:tasks.rados.rados.0.smithi170.stdout:776: left oid 22 (ObjNum 372 snap 91 seq_num 372) 2024-03-27T09:38:22.770 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:22.768+0000 7f458a0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:22.774 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:22.773+0000 7f458a0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:22.787 INFO:tasks.rados.rados.0.smithi170.stdout:778: finishing write tid 2 to smithi17031214-39 2024-03-27T09:38:22.791 INFO:tasks.rados.rados.0.smithi170.stdout:769: finishing write tid 2 to smithi17031214-26 2024-03-27T09:38:22.798 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:22.799 INFO:tasks.rados.rados.0.smithi170.stdout:764: finishing write tid 1 to smithi17031214-34 2024-03-27T09:38:22.802 INFO:tasks.rados.rados.0.smithi170.stdout:767: finishing write tid 2 to smithi17031214-7 2024-03-27T09:38:22.806 INFO:tasks.rados.rados.0.smithi170.stdout:772: expect (ObjNum 336 snap 83 seq_num 336) 2024-03-27T09:38:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:22.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:766: done (10 left) 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:771: done (9 left) 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:772: done (8 left) 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:776: done (7 left) 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:780: done (6 left) 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:781: delete oid 19 current snap is 91 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:765: finishing write tid 2 to smithi17031214-10 2024-03-27T09:38:23.039 INFO:tasks.rados.rados.0.smithi170.stdout:778: finishing write tid 3 to smithi17031214-39 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: finishing write tid 3 to smithi17031214-7 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: finishing write tid 4 to smithi17031214-7 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: oid 7 updating version 0 to 89 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: oid 7 updating version 89 to 91 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: oid 7 version 91 is already newer than 91 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 91 (ObjNum 368 snap 91 seq_num 368) dirty exists 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:767: left oid 7 (ObjNum 368 snap 91 seq_num 368) 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:769: finishing write tid 3 to smithi17031214-26 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:778: finishing write tid 4 to smithi17031214-39 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:778: oid 39 updating version 0 to 142 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:778: oid 39 updating version 142 to 144 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:778: oid 39 version 144 is already newer than 144 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 144 (ObjNum 374 snap 91 seq_num 374) dirty exists 2024-03-27T09:38:23.040 INFO:tasks.rados.rados.0.smithi170.stdout:778: left oid 39 (ObjNum 374 snap 91 seq_num 374) 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:769: finishing write tid 4 to smithi17031214-26 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:769: oid 26 updating version 0 to 234 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:769: oid 26 version 234 is already newer than 233 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:769: oid 26 version 234 is already newer than 232 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 234 (ObjNum 369 snap 91 seq_num 369) dirty exists 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:769: left oid 26 (ObjNum 369 snap 91 seq_num 369) 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:764: finishing write tid 2 to smithi17031214-34 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: finishing write tid 3 to smithi17031214-10 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: finishing write tid 4 to smithi17031214-10 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: oid 10 updating version 0 to 37 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: oid 10 version 37 is already newer than 35 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: oid 10 version 37 is already newer than 36 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 37 (ObjNum 366 snap 91 seq_num 366) dirty exists 2024-03-27T09:38:23.041 INFO:tasks.rados.rados.0.smithi170.stdout:765: left oid 10 (ObjNum 366 snap 91 seq_num 366) 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: finishing write tid 3 to smithi17031214-34 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: finishing write tid 1 to smithi17031214-31 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: finishing write tid 4 to smithi17031214-34 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: oid 34 updating version 0 to 360 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: oid 34 version 360 is already newer than 358 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: oid 34 version 360 is already newer than 360 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 360 (ObjNum 365 snap 91 seq_num 365) dirty exists 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:764: left oid 34 (ObjNum 365 snap 91 seq_num 365) 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: finishing write tid 2 to smithi17031214-31 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: finishing write tid 3 to smithi17031214-31 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: finishing write tid 4 to smithi17031214-31 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: oid 31 updating version 0 to 294 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: oid 31 version 294 is already newer than 292 2024-03-27T09:38:23.042 INFO:tasks.rados.rados.0.smithi170.stdout:777: oid 31 version 294 is already newer than 294 2024-03-27T09:38:23.043 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 294 (ObjNum 373 snap 91 seq_num 373) dirty exists 2024-03-27T09:38:23.043 INFO:tasks.rados.rados.0.smithi170.stdout:777: left oid 31 (ObjNum 373 snap 91 seq_num 373) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:764: done (6 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:765: done (5 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:767: done (4 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:769: done (3 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:777: done (2 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:778: done (1 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:781: done (0 left) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:782: read oid 36 snap -1 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:782: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:783: write oid 24 current snap is 91 2024-03-27T09:38:23.058 INFO:tasks.rados.rados.0.smithi170.stdout:783: seq_num 378 ranges {753046=545419,1786664=431793} 2024-03-27T09:38:23.065 INFO:tasks.rados.rados.0.smithi170.stdout:783: writing smithi17031214-24 from 753046 to 1298465 tid 1 2024-03-27T09:38:23.070 INFO:tasks.rados.rados.0.smithi170.stdout:783: writing smithi17031214-24 from 1786664 to 2218457 tid 2 2024-03-27T09:38:23.070 INFO:tasks.rados.rados.0.smithi170.stdout:784: read oid 28 snap -1 2024-03-27T09:38:23.070 INFO:tasks.rados.rados.0.smithi170.stdout:784: expect (ObjNum 331 snap 83 seq_num 331) 2024-03-27T09:38:23.070 INFO:tasks.rados.rados.0.smithi170.stdout:785: snap_create 2024-03-27T09:38:23.077 INFO:tasks.rados.rados.0.smithi170.stdout:782: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:783: finishing write tid 1 to smithi17031214-24 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:782: done (3 left) 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:785: done (2 left) 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:786: snap_create 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:783: finishing write tid 2 to smithi17031214-24 2024-03-27T09:38:23.345 INFO:tasks.rados.rados.0.smithi170.stdout:784: expect (ObjNum 331 snap 83 seq_num 331) 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: finishing write tid 3 to smithi17031214-24 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: finishing write tid 4 to smithi17031214-24 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: oid 24 updating version 0 to 135 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: oid 24 version 135 is already newer than 135 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: oid 24 version 135 is already newer than 133 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 135 (ObjNum 378 snap 91 seq_num 378) dirty exists 2024-03-27T09:38:23.587 INFO:tasks.rados.rados.0.smithi170.stdout:783: left oid 24 (ObjNum 378 snap 91 seq_num 378) 2024-03-27T09:38:24.162 INFO:tasks.rados.rados.0.smithi170.stdout:783: done (2 left) 2024-03-27T09:38:24.162 INFO:tasks.rados.rados.0.smithi170.stdout:784: done (1 left) 2024-03-27T09:38:24.163 INFO:tasks.rados.rados.0.smithi170.stdout:786: done (0 left) 2024-03-27T09:38:24.163 INFO:tasks.rados.rados.0.smithi170.stdout:787: snap_remove snap 91 2024-03-27T09:38:24.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:24.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2024-03-27T09:38:24.552 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:24.555 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:24.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:25.006 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:25.004+0000 7f40e842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:25.010 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:25.009+0000 7f40e842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:25.033 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:25.082 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:25.163 INFO:tasks.rados.rados.0.smithi170.stdout:787: done (0 left) 2024-03-27T09:38:25.164 INFO:tasks.rados.rados.0.smithi170.stdout:788: write oid 9 current snap is 93 2024-03-27T09:38:25.164 INFO:tasks.rados.rados.0.smithi170.stdout:788: seq_num 379 ranges {769853=620668,1845146=524707} 2024-03-27T09:38:25.171 INFO:tasks.rados.rados.0.smithi170.stdout:788: writing smithi17031214-9 from 769853 to 1390521 tid 1 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:788: writing smithi17031214-9 from 1845146 to 2369853 tid 2 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:789: read oid 33 snap -1 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:789: expect (ObjNum 332 snap 83 seq_num 332) 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:790: read oid 31 snap -1 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:790: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:791: write oid 29 current snap is 93 2024-03-27T09:38:25.177 INFO:tasks.rados.rados.0.smithi170.stdout:791: seq_num 380 ranges {786660=695917,1903628=435462} 2024-03-27T09:38:25.185 INFO:tasks.rados.rados.0.smithi170.stdout:791: writing smithi17031214-29 from 786660 to 1482577 tid 1 2024-03-27T09:38:25.192 INFO:tasks.rados.rados.0.smithi170.stdout:791: writing smithi17031214-29 from 1903628 to 2339090 tid 2 2024-03-27T09:38:25.192 INFO:tasks.rados.rados.0.smithi170.stdout:792: delete oid 48 current snap is 93 2024-03-27T09:38:25.241 INFO:tasks.rados.rados.0.smithi170.stdout:790: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-27T09:38:25.334 INFO:tasks.rados.rados.0.smithi170.stdout:788: finishing write tid 1 to smithi17031214-9 2024-03-27T09:38:25.334 INFO:tasks.rados.rados.0.smithi170.stdout:789: expect (ObjNum 332 snap 83 seq_num 332) 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: finishing write tid 2 to smithi17031214-9 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: finishing write tid 3 to smithi17031214-9 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: finishing write tid 4 to smithi17031214-9 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: oid 9 updating version 0 to 364 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: oid 9 version 364 is already newer than 364 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:788: oid 9 version 364 is already newer than 362 2024-03-27T09:38:25.511 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 364 (ObjNum 379 snap 93 seq_num 379) dirty exists 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:788: left oid 9 (ObjNum 379 snap 93 seq_num 379) 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:791: finishing write tid 1 to smithi17031214-29 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:788: done (4 left) 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:789: done (3 left) 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:790: done (2 left) 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:792: done (1 left) 2024-03-27T09:38:25.512 INFO:tasks.rados.rados.0.smithi170.stdout:793: snap_remove snap 92 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:793: done (1 left) 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:794: read oid 14 snap -1 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:794: expect deleted 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:795: write oid 45 current snap is 93 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:791: finishing write tid 2 to smithi17031214-29 2024-03-27T09:38:25.969 INFO:tasks.rados.rados.0.smithi170.stdout:791: finishing write tid 3 to smithi17031214-29 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:791: finishing write tid 4 to smithi17031214-29 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:791: oid 29 updating version 0 to 296 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:791: oid 29 updating version 296 to 298 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:791: oid 29 version 298 is already newer than 298 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 298 (ObjNum 380 snap 93 seq_num 380) dirty exists 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:791: left oid 29 (ObjNum 380 snap 93 seq_num 380) 2024-03-27T09:38:25.970 INFO:tasks.rados.rados.0.smithi170.stdout:795: seq_num 382 ranges {420274=562768,1620592=439131} 2024-03-27T09:38:25.974 INFO:tasks.rados.rados.0.smithi170.stdout:795: writing smithi17031214-45 from 420274 to 983042 tid 1 2024-03-27T09:38:25.981 INFO:tasks.rados.rados.0.smithi170.stdout:795: writing smithi17031214-45 from 1620592 to 2059723 tid 2 2024-03-27T09:38:25.981 INFO:tasks.rados.rados.0.smithi170.stdout:791: done (2 left) 2024-03-27T09:38:25.981 INFO:tasks.rados.rados.0.smithi170.stdout:796: write (excl) oid 7 current snap is 93 2024-03-27T09:38:25.981 INFO:tasks.rados.rados.0.smithi170.stdout:796: seq_num 383 ranges {437081=638017,1679074=582789} 2024-03-27T09:38:25.987 INFO:tasks.rados.rados.0.smithi170.stdout:796: writing smithi17031214-7 from 437081 to 1075098 tid 1 2024-03-27T09:38:25.996 INFO:tasks.rados.rados.0.smithi170.stdout:796: writing smithi17031214-7 from 1679074 to 2261863 tid 2 2024-03-27T09:38:25.997 INFO:tasks.rados.rados.0.smithi170.stdout:797: delete oid 42 current snap is 93 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:794: done (3 left) 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:797: done (2 left) 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:798: read oid 20 snap -1 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:798: expect deleted 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:799: read oid 26 snap -1 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:799: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-27T09:38:26.025 INFO:tasks.rados.rados.0.smithi170.stdout:800: snap_create 2024-03-27T09:38:26.078 INFO:tasks.rados.rados.0.smithi170.stdout:795: finishing write tid 1 to smithi17031214-45 2024-03-27T09:38:26.101 INFO:tasks.rados.rados.0.smithi170.stdout:796: finishing write tid 1 to smithi17031214-7 2024-03-27T09:38:26.112 INFO:tasks.rados.rados.0.smithi170.stdout:795: finishing write tid 2 to smithi17031214-45 2024-03-27T09:38:26.115 INFO:tasks.rados.rados.0.smithi170.stdout:799: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-27T09:38:26.243 INFO:tasks.rados.rados.0.smithi170.stdout:796: finishing write tid 2 to smithi17031214-7 2024-03-27T09:38:26.243 INFO:tasks.rados.rados.0.smithi170.stdout:795: finishing write tid 3 to smithi17031214-45 2024-03-27T09:38:26.243 INFO:tasks.rados.rados.0.smithi170.stdout:795: finishing write tid 4 to smithi17031214-45 2024-03-27T09:38:26.243 INFO:tasks.rados.rados.0.smithi170.stdout:795: oid 45 updating version 0 to 300 2024-03-27T09:38:26.243 INFO:tasks.rados.rados.0.smithi170.stdout:795: oid 45 updating version 300 to 302 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:795: oid 45 version 302 is already newer than 301 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 302 (ObjNum 382 snap 93 seq_num 382) dirty exists 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:795: left oid 45 (ObjNum 382 snap 93 seq_num 382) 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: finishing write tid 3 to smithi17031214-7 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: finishing write tid 4 to smithi17031214-7 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: oid 7 updating version 0 to 94 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: oid 7 version 94 is already newer than 92 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: oid 7 version 94 is already newer than 93 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 94 (ObjNum 383 snap 93 seq_num 383) dirty exists 2024-03-27T09:38:26.244 INFO:tasks.rados.rados.0.smithi170.stdout:796: left oid 7 (ObjNum 383 snap 93 seq_num 383) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:795: done (4 left) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:796: done (3 left) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:798: done (2 left) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:799: done (1 left) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:800: done (0 left) 2024-03-27T09:38:26.970 INFO:tasks.rados.rados.0.smithi170.stdout:801: snap_remove snap 93 2024-03-27T09:38:27.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:27.196 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:27.194+0000 7f88e5b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:27.200 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:27.199+0000 7f88e5b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:27.223 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:27.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:801: done (0 left) 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:802: read oid 29 snap -1 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:802: expect (ObjNum 380 snap 93 seq_num 380) 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:803: read oid 3 snap -1 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:803: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-27T09:38:27.971 INFO:tasks.rados.rados.0.smithi170.stdout:804: snap_create 2024-03-27T09:38:28.035 INFO:tasks.rados.rados.0.smithi170.stdout:803: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-27T09:38:28.116 INFO:tasks.rados.rados.0.smithi170.stdout:802: expect (ObjNum 380 snap 93 seq_num 380) 2024-03-27T09:38:28.971 INFO:tasks.rados.rados.0.smithi170.stdout:802: done (2 left) 2024-03-27T09:38:28.971 INFO:tasks.rados.rados.0.smithi170.stdout:803: done (1 left) 2024-03-27T09:38:28.972 INFO:tasks.rados.rados.0.smithi170.stdout:804: done (0 left) 2024-03-27T09:38:28.972 INFO:tasks.rados.rados.0.smithi170.stdout:805: delete oid 11 current snap is 95 2024-03-27T09:38:29.043 INFO:tasks.rados.rados.0.smithi170.stdout:805: done (0 left) 2024-03-27T09:38:29.043 INFO:tasks.rados.rados.0.smithi170.stdout:806: snap_remove snap 94 2024-03-27T09:38:29.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:29.380 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:29.378+0000 7fe1e646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:29.384 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:29.383+0000 7fe1e646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:29.408 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:29.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:29.556 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:29.656 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:29.659 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:29.972 INFO:tasks.rados.rados.0.smithi170.stdout:806: done (0 left) 2024-03-27T09:38:29.973 INFO:tasks.rados.rados.0.smithi170.stdout:807: read oid 44 snap -1 2024-03-27T09:38:29.973 INFO:tasks.rados.rados.0.smithi170.stdout:807: expect (ObjNum 356 snap 90 seq_num 356) 2024-03-27T09:38:29.973 INFO:tasks.rados.rados.0.smithi170.stdout:808: write oid 16 current snap is 95 2024-03-27T09:38:29.973 INFO:tasks.rados.rados.0.smithi170.stdout:808: seq_num 386 ranges {487502=463764,1738167=446469} 2024-03-27T09:38:29.977 INFO:tasks.rados.rados.0.smithi170.stdout:808: writing smithi17031214-16 from 487502 to 951266 tid 1 2024-03-27T09:38:29.985 INFO:tasks.rados.rados.0.smithi170.stdout:808: writing smithi17031214-16 from 1738167 to 2184636 tid 2 2024-03-27T09:38:29.985 INFO:tasks.rados.rados.0.smithi170.stdout:809: read oid 24 snap -1 2024-03-27T09:38:29.985 INFO:tasks.rados.rados.0.smithi170.stdout:809: expect (ObjNum 378 snap 91 seq_num 378) 2024-03-27T09:38:29.985 INFO:tasks.rados.rados.0.smithi170.stdout:810: write oid 19 current snap is 95 2024-03-27T09:38:29.985 INFO:tasks.rados.rados.0.smithi170.stdout:810: seq_num 387 ranges {504309=539013,1796649=590127} 2024-03-27T09:38:29.994 INFO:tasks.rados.rados.0.smithi170.stdout:810: writing smithi17031214-19 from 504309 to 1043322 tid 1 2024-03-27T09:38:30.003 INFO:tasks.rados.rados.0.smithi170.stdout:810: writing smithi17031214-19 from 1796649 to 2386776 tid 2 2024-03-27T09:38:30.004 INFO:tasks.rados.rados.0.smithi170.stdout:811: read oid 42 snap -1 2024-03-27T09:38:30.004 INFO:tasks.rados.rados.0.smithi170.stdout:811: expect deleted 2024-03-27T09:38:30.004 INFO:tasks.rados.rados.0.smithi170.stdout:812: write (excl) oid 11 current snap is 95 2024-03-27T09:38:30.004 INFO:tasks.rados.rados.0.smithi170.stdout:812: seq_num 388 ranges {521116=730615,1971484=549632} 2024-03-27T09:38:30.013 INFO:tasks.rados.rados.0.smithi170.stdout:812: writing smithi17031214-11 from 521116 to 1251731 tid 1 2024-03-27T09:38:30.022 INFO:tasks.rados.rados.0.smithi170.stdout:812: writing smithi17031214-11 from 1971484 to 2521116 tid 2 2024-03-27T09:38:30.023 INFO:tasks.rados.rados.0.smithi170.stdout:813: delete oid 13 current snap is 95 2024-03-27T09:38:30.032 INFO:tasks.rados.rados.0.smithi170.stdout:811: done (6 left) 2024-03-27T09:38:30.032 INFO:tasks.rados.rados.0.smithi170.stdout:813: done (5 left) 2024-03-27T09:38:30.033 INFO:tasks.rados.rados.0.smithi170.stdout:814: delete oid 45 current snap is 95 2024-03-27T09:38:30.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2024-03-27T09:38:30.066 INFO:tasks.rados.rados.0.smithi170.stdout:809: expect (ObjNum 378 snap 91 seq_num 378) 2024-03-27T09:38:30.158 INFO:tasks.rados.rados.0.smithi170.stdout:807: expect (ObjNum 356 snap 90 seq_num 356) 2024-03-27T09:38:30.359 INFO:tasks.rados.rados.0.smithi170.stdout:810: finishing write tid 1 to smithi17031214-19 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:808: finishing write tid 1 to smithi17031214-16 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:812: finishing write tid 1 to smithi17031214-11 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: finishing write tid 2 to smithi17031214-19 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:812: finishing write tid 2 to smithi17031214-11 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:808: finishing write tid 2 to smithi17031214-16 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: finishing write tid 3 to smithi17031214-19 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: finishing write tid 4 to smithi17031214-19 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: oid 19 updating version 0 to 238 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: oid 19 version 238 is already newer than 236 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: oid 19 version 238 is already newer than 237 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 238 (ObjNum 387 snap 95 seq_num 387) dirty exists 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:810: left oid 19 (ObjNum 387 snap 95 seq_num 387) 2024-03-27T09:38:30.360 INFO:tasks.rados.rados.0.smithi170.stdout:812: finishing write tid 3 to smithi17031214-11 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:807: done (5 left) 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:809: done (4 left) 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:810: done (3 left) 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:814: done (2 left) 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:815: read oid 27 snap -1 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:815: expect deleted 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:816: delete oid 17 current snap is 95 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:808: finishing write tid 3 to smithi17031214-16 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:812: finishing write tid 4 to smithi17031214-11 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:812: oid 11 updating version 0 to 307 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:812: oid 11 version 307 is already newer than 305 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:812: oid 11 version 307 is already newer than 306 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 307 (ObjNum 388 snap 95 seq_num 388) dirty exists 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:812: left oid 11 (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:808: finishing write tid 4 to smithi17031214-16 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:808: oid 16 updating version 0 to 96 2024-03-27T09:38:30.361 INFO:tasks.rados.rados.0.smithi170.stdout:808: oid 16 updating version 96 to 97 2024-03-27T09:38:30.362 INFO:tasks.rados.rados.0.smithi170.stdout:808: oid 16 version 97 is already newer than 95 2024-03-27T09:38:30.362 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 97 (ObjNum 386 snap 95 seq_num 386) dirty exists 2024-03-27T09:38:30.362 INFO:tasks.rados.rados.0.smithi170.stdout:808: left oid 16 (ObjNum 386 snap 95 seq_num 386) 2024-03-27T09:38:30.391 INFO:tasks.rados.rados.0.smithi170.stdout:808: done (3 left) 2024-03-27T09:38:30.391 INFO:tasks.rados.rados.0.smithi170.stdout:812: done (2 left) 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:815: done (1 left) 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:816: done (0 left) 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:817: read oid 26 snap -1 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:817: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:818: write (excl) oid 16 current snap is 95 2024-03-27T09:38:30.392 INFO:tasks.rados.rados.0.smithi170.stdout:818: seq_num 392 ranges {588344=631611,1689059=741123} 2024-03-27T09:38:30.401 INFO:tasks.rados.rados.0.smithi170.stdout:818: writing smithi17031214-16 from 588344 to 1219955 tid 1 2024-03-27T09:38:30.407 INFO:tasks.rados.rados.0.smithi170.stdout:818: writing smithi17031214-16 from 1689059 to 2430182 tid 2 2024-03-27T09:38:30.408 INFO:tasks.rados.rados.0.smithi170.stdout:819: snap_create 2024-03-27T09:38:30.408 INFO:tasks.rados.rados.0.smithi170.stdout:817: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-27T09:38:30.534 INFO:tasks.rados.rados.0.smithi170.stdout:818: finishing write tid 1 to smithi17031214-16 2024-03-27T09:38:30.534 INFO:tasks.rados.rados.0.smithi170.stdout:818: finishing write tid 2 to smithi17031214-16 2024-03-27T09:38:30.534 INFO:tasks.rados.rados.0.smithi170.stdout:818: finishing write tid 3 to smithi17031214-16 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:818: finishing write tid 4 to smithi17031214-16 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:818: oid 16 updating version 0 to 98 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:818: oid 16 updating version 98 to 99 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:818: oid 16 updating version 99 to 100 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 100 (ObjNum 392 snap 95 seq_num 392) dirty exists 2024-03-27T09:38:30.535 INFO:tasks.rados.rados.0.smithi170.stdout:818: left oid 16 (ObjNum 392 snap 95 seq_num 392) 2024-03-27T09:38:31.181 INFO:tasks.rados.rados.0.smithi170.stdout:817: done (2 left) 2024-03-27T09:38:31.181 INFO:tasks.rados.rados.0.smithi170.stdout:818: done (1 left) 2024-03-27T09:38:31.181 INFO:tasks.rados.rados.0.smithi170.stdout:819: done (0 left) 2024-03-27T09:38:31.181 INFO:tasks.rados.rados.0.smithi170.stdout:820: delete oid 23 current snap is 96 2024-03-27T09:38:31.220 INFO:tasks.rados.rados.0.smithi170.stdout:820: done (0 left) 2024-03-27T09:38:31.220 INFO:tasks.rados.rados.0.smithi170.stdout:821: delete oid 27 current snap is 96 2024-03-27T09:38:31.234 INFO:tasks.rados.rados.0.smithi170.stdout:821: done (0 left) 2024-03-27T09:38:31.234 INFO:tasks.rados.rados.0.smithi170.stdout:822: delete oid 18 current snap is 96 2024-03-27T09:38:31.258 INFO:tasks.rados.rados.0.smithi170.stdout:822: done (0 left) 2024-03-27T09:38:31.259 INFO:tasks.rados.rados.0.smithi170.stdout:823: delete oid 39 current snap is 96 2024-03-27T09:38:31.315 INFO:tasks.rados.rados.0.smithi170.stdout:823: done (0 left) 2024-03-27T09:38:31.315 INFO:tasks.rados.rados.0.smithi170.stdout:824: snap_remove snap 95 2024-03-27T09:38:31.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:31.563 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:31.561+0000 7f46be55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:31.567 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:31.566+0000 7f46be55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:31.591 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:31.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:32.185 INFO:tasks.rados.rados.0.smithi170.stdout:824: done (0 left) 2024-03-27T09:38:32.185 INFO:tasks.rados.rados.0.smithi170.stdout:825: read oid 6 snap -1 2024-03-27T09:38:32.185 INFO:tasks.rados.rados.0.smithi170.stdout:825: expect deleted 2024-03-27T09:38:32.185 INFO:tasks.rados.rados.0.smithi170.stdout:826: delete oid 47 current snap is 96 2024-03-27T09:38:32.262 INFO:tasks.rados.rados.0.smithi170.stdout:825: done (1 left) 2024-03-27T09:38:32.262 INFO:tasks.rados.rados.0.smithi170.stdout:826: done (0 left) 2024-03-27T09:38:32.263 INFO:tasks.rados.rados.0.smithi170.stdout:827: write oid 5 current snap is 96 2024-03-27T09:38:32.263 INFO:tasks.rados.rados.0.smithi170.stdout:827: seq_num 398 ranges {689186=799458,1923598=752130} 2024-03-27T09:38:32.270 INFO:tasks.rados.rados.0.smithi170.stdout:827: writing smithi17031214-5 from 689186 to 1488644 tid 1 2024-03-27T09:38:32.276 INFO:tasks.rados.rados.0.smithi170.stdout:827: writing smithi17031214-5 from 1923598 to 2675728 tid 2 2024-03-27T09:38:32.276 INFO:tasks.rados.rados.0.smithi170.stdout:828: snap_create 2024-03-27T09:38:32.294 INFO:tasks.rados.rados.0.smithi170.stdout:827: finishing write tid 1 to smithi17031214-5 2024-03-27T09:38:32.313 INFO:tasks.rados.rados.0.smithi170.stdout:827: finishing write tid 2 to smithi17031214-5 2024-03-27T09:38:32.320 INFO:tasks.rados.rados.0.smithi170.stdout:827: finishing write tid 3 to smithi17031214-5 2024-03-27T09:38:32.321 INFO:tasks.rados.rados.0.smithi170.stdout:827: finishing write tid 4 to smithi17031214-5 2024-03-27T09:38:32.322 INFO:tasks.rados.rados.0.smithi170.stdout:827: oid 5 updating version 0 to 151 2024-03-27T09:38:32.322 INFO:tasks.rados.rados.0.smithi170.stdout:827: oid 5 updating version 151 to 152 2024-03-27T09:38:32.322 INFO:tasks.rados.rados.0.smithi170.stdout:827: oid 5 version 152 is already newer than 150 2024-03-27T09:38:32.322 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 152 (ObjNum 398 snap 96 seq_num 398) dirty exists 2024-03-27T09:38:32.322 INFO:tasks.rados.rados.0.smithi170.stdout:827: left oid 5 (ObjNum 398 snap 96 seq_num 398) 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:827: done (1 left) 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:828: done (0 left) 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:829: read oid 3 snap 96 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:829: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:830: write oid 49 current snap is 97 2024-03-27T09:38:33.188 INFO:tasks.rados.rados.0.smithi170.stdout:830: seq_num 399 ranges {705993=474707,1582080=612141} 2024-03-27T09:38:33.194 INFO:tasks.rados.rados.0.smithi170.stdout:830: writing smithi17031214-49 from 705993 to 1180700 tid 1 2024-03-27T09:38:33.201 INFO:tasks.rados.rados.0.smithi170.stdout:830: writing smithi17031214-49 from 1582080 to 2194221 tid 2 2024-03-27T09:38:33.201 INFO:tasks.rados.rados.0.smithi170.stdout:831: read oid 17 snap -1 2024-03-27T09:38:33.201 INFO:tasks.rados.rados.0.smithi170.stdout:831: expect deleted 2024-03-27T09:38:33.201 INFO:tasks.rados.rados.0.smithi170.stdout:832: snap_create 2024-03-27T09:38:33.253 INFO:tasks.rados.rados.0.smithi170.stdout:829: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-27T09:38:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:830: finishing write tid 1 to smithi17031214-49 2024-03-27T09:38:33.336 INFO:tasks.rados.rados.0.smithi170.stdout:830: finishing write tid 2 to smithi17031214-49 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: finishing write tid 3 to smithi17031214-49 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: finishing write tid 4 to smithi17031214-49 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: oid 49 updating version 0 to 206 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: oid 49 updating version 206 to 208 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: oid 49 version 208 is already newer than 207 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 208 (ObjNum 399 snap 97 seq_num 399) dirty exists 2024-03-27T09:38:33.337 INFO:tasks.rados.rados.0.smithi170.stdout:830: left oid 49 (ObjNum 399 snap 97 seq_num 399) 2024-03-27T09:38:33.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:33.745 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:33.742+0000 7f1bbd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:33.749 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:33.748+0000 7f1bbd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:33.773 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:33.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:34.191 INFO:tasks.rados.rados.0.smithi170.stdout:829: done (3 left) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:830: done (2 left) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:831: done (1 left) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:832: done (0 left) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:833: read oid 1 snap -1 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:833: expect deleted 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:834: read oid 43 snap -1 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:834: expect (ObjNum 344 snap 86 seq_num 344) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:835: read oid 41 snap -1 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:835: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:836: write (excl) oid 25 current snap is 98 2024-03-27T09:38:34.192 INFO:tasks.rados.rados.0.smithi170.stdout:836: seq_num 400 ranges {722800=549956,2040562=682238} 2024-03-27T09:38:34.198 INFO:tasks.rados.rados.0.smithi170.stdout:836: writing smithi17031214-25 from 722800 to 1272756 tid 1 2024-03-27T09:38:34.207 INFO:tasks.rados.rados.0.smithi170.stdout:836: writing smithi17031214-25 from 2040562 to 2722800 tid 2 2024-03-27T09:38:34.207 INFO:tasks.rados.rados.0.smithi170.stdout:837: delete oid 13 current snap is 98 2024-03-27T09:38:34.233 INFO:tasks.rados.rados.0.smithi170.stdout:837: done (4 left) 2024-03-27T09:38:34.233 INFO:tasks.rados.rados.0.smithi170.stdout:838: read oid 8 snap 96 2024-03-27T09:38:34.233 INFO:tasks.rados.rados.0.smithi170.stdout:838: expect deleted 2024-03-27T09:38:34.233 INFO:tasks.rados.rados.0.smithi170.stdout:839: read oid 21 snap -1 2024-03-27T09:38:34.234 INFO:tasks.rados.rados.0.smithi170.stdout:839: expect deleted 2024-03-27T09:38:34.234 INFO:tasks.rados.rados.0.smithi170.stdout:840: read oid 13 snap 97 2024-03-27T09:38:34.234 INFO:tasks.rados.rados.0.smithi170.stdout:840: expect deleted 2024-03-27T09:38:34.234 INFO:tasks.rados.rados.0.smithi170.stdout:841: delete oid 45 current snap is 98 2024-03-27T09:38:34.248 INFO:tasks.rados.rados.0.smithi170.stdout:835: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:38:34.365 INFO:tasks.rados.rados.0.smithi170.stdout:833: done (7 left) 2024-03-27T09:38:34.365 INFO:tasks.rados.rados.0.smithi170.stdout:835: done (6 left) 2024-03-27T09:38:34.365 INFO:tasks.rados.rados.0.smithi170.stdout:840: done (5 left) 2024-03-27T09:38:34.365 INFO:tasks.rados.rados.0.smithi170.stdout:841: done (4 left) 2024-03-27T09:38:34.365 INFO:tasks.rados.rados.0.smithi170.stdout:842: snap_remove snap 97 2024-03-27T09:38:34.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:34.759 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:34.762 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:35.194 INFO:tasks.rados.rados.0.smithi170.stdout:842: done (4 left) 2024-03-27T09:38:35.194 INFO:tasks.rados.rados.0.smithi170.stdout:843: write oid 39 current snap is 98 2024-03-27T09:38:35.194 INFO:tasks.rados.rados.0.smithi170.stdout:843: seq_num 403 ranges {773221=492056,1816008=503126} 2024-03-27T09:38:35.201 INFO:tasks.rados.rados.0.smithi170.stdout:843: writing smithi17031214-39 from 773221 to 1265277 tid 1 2024-03-27T09:38:35.207 INFO:tasks.rados.rados.0.smithi170.stdout:843: writing smithi17031214-39 from 1816008 to 2319134 tid 2 2024-03-27T09:38:35.207 INFO:tasks.rados.rados.0.smithi170.stdout:844: read oid 31 snap 96 2024-03-27T09:38:35.207 INFO:tasks.rados.rados.0.smithi170.stdout:844: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-27T09:38:35.207 INFO:tasks.rados.rados.0.smithi170.stdout:845: write (excl) oid 15 current snap is 98 2024-03-27T09:38:35.207 INFO:tasks.rados.rados.0.smithi170.stdout:834: expect (ObjNum 344 snap 86 seq_num 344) 2024-03-27T09:38:35.366 INFO:tasks.rados.rados.0.smithi170.stdout:836: finishing write tid 1 to smithi17031214-25 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: finishing write tid 2 to smithi17031214-25 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: finishing write tid 3 to smithi17031214-25 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: finishing write tid 4 to smithi17031214-25 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: oid 25 updating version 0 to 306 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: oid 25 updating version 306 to 308 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: oid 25 version 308 is already newer than 307 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 308 (ObjNum 400 snap 98 seq_num 400) dirty exists 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:836: left oid 25 (ObjNum 400 snap 98 seq_num 400) 2024-03-27T09:38:35.367 INFO:tasks.rados.rados.0.smithi170.stdout:845: seq_num 404 ranges {790028=567305,1874490=763137} 2024-03-27T09:38:35.373 INFO:tasks.rados.rados.0.smithi170.stdout:845: writing smithi17031214-15 from 790028 to 1357333 tid 1 2024-03-27T09:38:35.381 INFO:tasks.rados.rados.0.smithi170.stdout:845: writing smithi17031214-15 from 1874490 to 2637627 tid 2 2024-03-27T09:38:35.381 INFO:tasks.rados.rados.0.smithi170.stdout:834: done (6 left) 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:836: done (5 left) 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:838: done (4 left) 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:839: done (3 left) 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:846: read oid 43 snap 96 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:846: expect (ObjNum 344 snap 86 seq_num 344) 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:847: write (excl) oid 30 current snap is 98 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:843: finishing write tid 1 to smithi17031214-39 2024-03-27T09:38:35.382 INFO:tasks.rados.rados.0.smithi170.stdout:847: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-03-27T09:38:35.387 INFO:tasks.rados.rados.0.smithi170.stdout:847: writing smithi17031214-30 from 406835 to 1049389 tid 1 2024-03-27T09:38:35.392 INFO:tasks.rados.rados.0.smithi170.stdout:847: writing smithi17031214-30 from 1532972 to 2039767 tid 2 2024-03-27T09:38:35.397 INFO:tasks.rados.rados.0.smithi170.stdout:847: writing smithi17031214-30 from 2772312 to 2806835 tid 3 2024-03-27T09:38:35.397 INFO:tasks.rados.rados.0.smithi170.stdout:848: delete oid 49 current snap is 98 2024-03-27T09:38:35.397 INFO:tasks.rados.rados.0.smithi170.stdout:844: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-27T09:38:35.500 INFO:tasks.rados.rados.0.smithi170.stdout:843: finishing write tid 2 to smithi17031214-39 2024-03-27T09:38:35.500 INFO:tasks.rados.rados.0.smithi170.stdout:843: finishing write tid 3 to smithi17031214-39 2024-03-27T09:38:35.500 INFO:tasks.rados.rados.0.smithi170.stdout:843: finishing write tid 4 to smithi17031214-39 2024-03-27T09:38:35.500 INFO:tasks.rados.rados.0.smithi170.stdout:843: oid 39 updating version 0 to 155 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:843: oid 39 version 155 is already newer than 153 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:843: oid 39 version 155 is already newer than 155 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 155 (ObjNum 403 snap 98 seq_num 403) dirty exists 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:843: left oid 39 (ObjNum 403 snap 98 seq_num 403) 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:843: done (5 left) 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:844: done (4 left) 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:848: done (3 left) 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:849: delete oid 44 current snap is 98 2024-03-27T09:38:35.501 INFO:tasks.rados.rados.0.smithi170.stdout:846: expect (ObjNum 344 snap 86 seq_num 344) 2024-03-27T09:38:35.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: finishing write tid 1 to smithi17031214-15 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:847: finishing write tid 1 to smithi17031214-30 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: finishing write tid 2 to smithi17031214-15 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:847: finishing write tid 2 to smithi17031214-30 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: finishing write tid 3 to smithi17031214-15 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: finishing write tid 4 to smithi17031214-15 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: oid 15 updating version 0 to 139 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: oid 15 version 139 is already newer than 138 2024-03-27T09:38:35.643 INFO:tasks.rados.rados.0.smithi170.stdout:845: oid 15 version 139 is already newer than 137 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 139 (ObjNum 404 snap 98 seq_num 404) dirty exists 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:845: left oid 15 (ObjNum 404 snap 98 seq_num 404) 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: finishing write tid 3 to smithi17031214-30 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:845: done (3 left) 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:846: done (2 left) 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:849: done (1 left) 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:850: delete oid 13 current snap is 98 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: finishing write tid 4 to smithi17031214-30 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: finishing write tid 5 to smithi17031214-30 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: oid 30 updating version 0 to 194 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: oid 30 updating version 194 to 195 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: oid 30 version 195 is already newer than 192 2024-03-27T09:38:35.644 INFO:tasks.rados.rados.0.smithi170.stdout:847: oid 30 version 195 is already newer than 194 2024-03-27T09:38:35.645 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 195 (ObjNum 405 snap 98 seq_num 405) dirty exists 2024-03-27T09:38:35.645 INFO:tasks.rados.rados.0.smithi170.stdout:847: left oid 30 (ObjNum 405 snap 98 seq_num 405) 2024-03-27T09:38:35.649 INFO:tasks.rados.rados.0.smithi170.stdout:847: done (1 left) 2024-03-27T09:38:35.649 INFO:tasks.rados.rados.0.smithi170.stdout:850: done (0 left) 2024-03-27T09:38:35.649 INFO:tasks.rados.rados.0.smithi170.stdout:851: read oid 22 snap -1 2024-03-27T09:38:35.649 INFO:tasks.rados.rados.0.smithi170.stdout:851: expect (ObjNum 372 snap 91 seq_num 372) 2024-03-27T09:38:35.650 INFO:tasks.rados.rados.0.smithi170.stdout:852: snap_remove snap 96 2024-03-27T09:38:35.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:35.938 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:35.933+0000 7fd4c36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:35.940 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:35.939+0000 7fd4c36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:35.963 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:35.970 INFO:tasks.rados.rados.0.smithi170.stdout:852: done (1 left) 2024-03-27T09:38:35.970 INFO:tasks.rados.rados.0.smithi170.stdout:853: delete oid 3 current snap is 98 2024-03-27T09:38:35.970 INFO:tasks.rados.rados.0.smithi170.stdout:851: expect (ObjNum 372 snap 91 seq_num 372) 2024-03-27T09:38:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:36.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:36.078 INFO:tasks.rados.rados.0.smithi170.stdout:851: done (1 left) 2024-03-27T09:38:36.078 INFO:tasks.rados.rados.0.smithi170.stdout:853: done (0 left) 2024-03-27T09:38:36.078 INFO:tasks.rados.rados.0.smithi170.stdout:854: delete oid 16 current snap is 98 2024-03-27T09:38:36.160 INFO:tasks.rados.rados.0.smithi170.stdout:854: done (0 left) 2024-03-27T09:38:36.160 INFO:tasks.rados.rados.0.smithi170.stdout:855: write oid 4 current snap is 98 2024-03-27T09:38:36.160 INFO:tasks.rados.rados.0.smithi170.stdout:855: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-03-27T09:38:36.165 INFO:tasks.rados.rados.0.smithi170.stdout:855: writing smithi17031214-4 from 507677 to 918078 tid 1 2024-03-27T09:38:36.170 INFO:tasks.rados.rados.0.smithi170.stdout:855: writing smithi17031214-4 from 1367511 to 1885313 tid 2 2024-03-27T09:38:36.176 INFO:tasks.rados.rados.0.smithi170.stdout:855: writing smithi17031214-4 from 2420497 to 2907677 tid 3 2024-03-27T09:38:36.176 INFO:tasks.rados.rados.0.smithi170.stdout:856: read oid 11 snap -1 2024-03-27T09:38:36.176 INFO:tasks.rados.rados.0.smithi170.stdout:856: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:36.177 INFO:tasks.rados.rados.0.smithi170.stdout:857: read oid 44 snap -1 2024-03-27T09:38:36.177 INFO:tasks.rados.rados.0.smithi170.stdout:857: expect deleted 2024-03-27T09:38:36.177 INFO:tasks.rados.rados.0.smithi170.stdout:858: delete oid 40 current snap is 98 2024-03-27T09:38:36.194 INFO:tasks.rados.rados.0.smithi170.stdout:855: finishing write tid 1 to smithi17031214-4 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:857: done (3 left) 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:858: done (2 left) 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:859: read oid 45 snap -1 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:859: expect deleted 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:860: write (excl) oid 3 current snap is 98 2024-03-27T09:38:36.224 INFO:tasks.rados.rados.0.smithi170.stdout:860: seq_num 413 ranges {541291=560899,1884475=521471} 2024-03-27T09:38:36.231 INFO:tasks.rados.rados.0.smithi170.stdout:860: writing smithi17031214-3 from 541291 to 1102190 tid 1 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:860: writing smithi17031214-3 from 1884475 to 2405946 tid 2 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:861: read oid 38 snap -1 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:861: expect deleted 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:862: snap_create 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:855: finishing write tid 2 to smithi17031214-4 2024-03-27T09:38:36.239 INFO:tasks.rados.rados.0.smithi170.stdout:856: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:36.340 INFO:tasks.rados.rados.0.smithi170.stdout:855: finishing write tid 3 to smithi17031214-4 2024-03-27T09:38:36.340 INFO:tasks.rados.rados.0.smithi170.stdout:855: finishing write tid 4 to smithi17031214-4 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: finishing write tid 1 to smithi17031214-3 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: finishing write tid 5 to smithi17031214-4 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: oid 4 updating version 0 to 36 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: oid 4 updating version 36 to 38 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: oid 4 version 38 is already newer than 35 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: oid 4 version 38 is already newer than 38 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 38 (ObjNum 411 snap 98 seq_num 411) dirty exists 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:855: left oid 4 (ObjNum 411 snap 98 seq_num 411) 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: finishing write tid 2 to smithi17031214-3 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: finishing write tid 3 to smithi17031214-3 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: finishing write tid 4 to smithi17031214-3 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: oid 3 updating version 0 to 213 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: oid 3 version 213 is already newer than 211 2024-03-27T09:38:36.341 INFO:tasks.rados.rados.0.smithi170.stdout:860: oid 3 version 213 is already newer than 213 2024-03-27T09:38:36.342 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 213 (ObjNum 413 snap 98 seq_num 413) dirty exists 2024-03-27T09:38:36.342 INFO:tasks.rados.rados.0.smithi170.stdout:860: left oid 3 (ObjNum 413 snap 98 seq_num 413) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:855: done (5 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:856: done (4 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:859: done (3 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:860: done (2 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:861: done (1 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:862: done (0 left) 2024-03-27T09:38:37.201 INFO:tasks.rados.rados.0.smithi170.stdout:863: delete oid 17 current snap is 99 2024-03-27T09:38:37.252 INFO:tasks.rados.rados.0.smithi170.stdout:863: done (0 left) 2024-03-27T09:38:37.252 INFO:tasks.rados.rados.0.smithi170.stdout:864: read oid 49 snap -1 2024-03-27T09:38:37.252 INFO:tasks.rados.rados.0.smithi170.stdout:864: expect deleted 2024-03-27T09:38:37.252 INFO:tasks.rados.rados.0.smithi170.stdout:865: delete oid 28 current snap is 99 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:864: done (1 left) 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:865: done (0 left) 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:866: read oid 11 snap -1 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:866: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:867: write oid 48 current snap is 99 2024-03-27T09:38:37.289 INFO:tasks.rados.rados.0.smithi170.stdout:867: seq_num 416 ranges {591712=786646,1943568=668798} 2024-03-27T09:38:37.296 INFO:tasks.rados.rados.0.smithi170.stdout:867: writing smithi17031214-48 from 591712 to 1378358 tid 1 2024-03-27T09:38:37.303 INFO:tasks.rados.rados.0.smithi170.stdout:867: writing smithi17031214-48 from 1943568 to 2612366 tid 2 2024-03-27T09:38:37.303 INFO:tasks.rados.rados.0.smithi170.stdout:868: write (excl) oid 27 current snap is 99 2024-03-27T09:38:37.303 INFO:tasks.rados.rados.0.smithi170.stdout:868: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-03-27T09:38:37.309 INFO:tasks.rados.rados.0.smithi170.stdout:868: writing smithi17031214-27 from 608519 to 1070414 tid 1 2024-03-27T09:38:37.315 INFO:tasks.rados.rados.0.smithi170.stdout:868: writing smithi17031214-27 from 1602050 to 2130859 tid 2 2024-03-27T09:38:37.320 INFO:tasks.rados.rados.0.smithi170.stdout:868: writing smithi17031214-27 from 2585035 to 3008519 tid 3 2024-03-27T09:38:37.320 INFO:tasks.rados.rados.0.smithi170.stdout:869: write oid 17 current snap is 99 2024-03-27T09:38:37.320 INFO:tasks.rados.rados.0.smithi170.stdout:869: seq_num 418 ranges {625326=537144,1944179=672467} 2024-03-27T09:38:37.326 INFO:tasks.rados.rados.0.smithi170.stdout:869: writing smithi17031214-17 from 625326 to 1162470 tid 1 2024-03-27T09:38:37.334 INFO:tasks.rados.rados.0.smithi170.stdout:869: writing smithi17031214-17 from 1944179 to 2616646 tid 2 2024-03-27T09:38:37.334 INFO:tasks.rados.rados.0.smithi170.stdout:870: write (excl) oid 4 current snap is 99 2024-03-27T09:38:37.335 INFO:tasks.rados.rados.0.smithi170.stdout:870: seq_num 419 ranges {642133=728746,2119014=532478} 2024-03-27T09:38:37.342 INFO:tasks.rados.rados.0.smithi170.stdout:870: writing smithi17031214-4 from 642133 to 1370879 tid 1 2024-03-27T09:38:37.349 INFO:tasks.rados.rados.0.smithi170.stdout:870: writing smithi17031214-4 from 2119014 to 2651492 tid 2 2024-03-27T09:38:37.350 INFO:tasks.rados.rados.0.smithi170.stdout:871: write (excl) oid 47 current snap is 99 2024-03-27T09:38:37.350 INFO:tasks.rados.rados.0.smithi170.stdout:871: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-03-27T09:38:37.355 INFO:tasks.rados.rados.0.smithi170.stdout:871: writing smithi17031214-47 from 658940 to 1062935 tid 1 2024-03-27T09:38:37.363 INFO:tasks.rados.rados.0.smithi170.stdout:871: writing smithi17031214-47 from 1777496 to 2453632 tid 2 2024-03-27T09:38:37.367 INFO:tasks.rados.rados.0.smithi170.stdout:871: writing smithi17031214-47 from 2950951 to 3058940 tid 3 2024-03-27T09:38:37.367 INFO:tasks.rados.rados.0.smithi170.stdout:872: write oid 3 current snap is 99 2024-03-27T09:38:37.367 INFO:tasks.rados.rados.0.smithi170.stdout:872: seq_num 421 ranges {675747=479244,1835978=536147} 2024-03-27T09:38:37.373 INFO:tasks.rados.rados.0.smithi170.stdout:872: writing smithi17031214-3 from 675747 to 1154991 tid 1 2024-03-27T09:38:37.380 INFO:tasks.rados.rados.0.smithi170.stdout:872: writing smithi17031214-3 from 1835978 to 2372125 tid 2 2024-03-27T09:38:37.380 INFO:tasks.rados.rados.0.smithi170.stdout:873: read oid 2 snap -1 2024-03-27T09:38:37.380 INFO:tasks.rados.rados.0.smithi170.stdout:873: expect deleted 2024-03-27T09:38:37.380 INFO:tasks.rados.rados.0.smithi170.stdout:874: write (excl) oid 50 current snap is 99 2024-03-27T09:38:37.380 INFO:tasks.rados.rados.0.smithi170.stdout:866: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:37.518 INFO:tasks.rados.rados.0.smithi170.stdout:867: finishing write tid 1 to smithi17031214-48 2024-03-27T09:38:37.518 INFO:tasks.rados.rados.0.smithi170.stdout:874: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-03-27T09:38:37.524 INFO:tasks.rados.rados.0.smithi170.stdout:874: writing smithi17031214-50 from 692554 to 1247047 tid 1 2024-03-27T09:38:37.531 INFO:tasks.rados.rados.0.smithi170.stdout:874: writing smithi17031214-50 from 1778107 to 2457912 tid 2 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:874: writing smithi17031214-50 from 2889444 to 3092554 tid 3 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:866: done (8 left) 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:875: snap_create 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:868: finishing write tid 1 to smithi17031214-27 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:867: finishing write tid 2 to smithi17031214-48 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:867: finishing write tid 3 to smithi17031214-48 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:867: finishing write tid 4 to smithi17031214-48 2024-03-27T09:38:37.536 INFO:tasks.rados.rados.0.smithi170.stdout:867: oid 48 updating version 0 to 215 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:867: oid 48 updating version 215 to 216 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:867: oid 48 version 216 is already newer than 214 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 216 (ObjNum 416 snap 99 seq_num 416) dirty exists 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:867: left oid 48 (ObjNum 416 snap 99 seq_num 416) 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:868: finishing write tid 2 to smithi17031214-27 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:869: finishing write tid 1 to smithi17031214-17 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:870: finishing write tid 1 to smithi17031214-4 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:872: finishing write tid 1 to smithi17031214-3 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:869: finishing write tid 2 to smithi17031214-17 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:868: finishing write tid 3 to smithi17031214-27 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:870: finishing write tid 2 to smithi17031214-4 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:868: finishing write tid 4 to smithi17031214-27 2024-03-27T09:38:37.537 INFO:tasks.rados.rados.0.smithi170.stdout:872: finishing write tid 2 to smithi17031214-3 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: finishing write tid 5 to smithi17031214-27 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: oid 27 updating version 0 to 316 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: oid 27 version 316 is already newer than 314 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: oid 27 version 316 is already newer than 315 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: oid 27 updating version 316 to 317 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 317 (ObjNum 417 snap 99 seq_num 417) dirty exists 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:868: left oid 27 (ObjNum 417 snap 99 seq_num 417) 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:869: finishing write tid 3 to smithi17031214-17 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:870: finishing write tid 3 to smithi17031214-4 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:869: finishing write tid 4 to smithi17031214-17 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:869: oid 17 updating version 0 to 306 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:869: oid 17 version 306 is already newer than 305 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:869: oid 17 updating version 306 to 307 2024-03-27T09:38:37.538 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 307 (ObjNum 418 snap 99 seq_num 418) dirty exists 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:869: left oid 17 (ObjNum 418 snap 99 seq_num 418) 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:870: finishing write tid 4 to smithi17031214-4 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:870: oid 4 updating version 0 to 42 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:870: oid 4 version 42 is already newer than 42 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:870: oid 4 version 42 is already newer than 40 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 42 (ObjNum 419 snap 99 seq_num 419) dirty exists 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:870: left oid 4 (ObjNum 419 snap 99 seq_num 419) 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:871: finishing write tid 1 to smithi17031214-47 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: finishing write tid 3 to smithi17031214-3 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: finishing write tid 4 to smithi17031214-3 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: oid 3 updating version 0 to 218 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: oid 3 updating version 218 to 220 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: oid 3 version 220 is already newer than 219 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 220 (ObjNum 421 snap 99 seq_num 421) dirty exists 2024-03-27T09:38:37.539 INFO:tasks.rados.rados.0.smithi170.stdout:872: left oid 3 (ObjNum 421 snap 99 seq_num 421) 2024-03-27T09:38:37.540 INFO:tasks.rados.rados.0.smithi170.stdout:871: finishing write tid 2 to smithi17031214-47 2024-03-27T09:38:37.540 INFO:tasks.rados.rados.0.smithi170.stdout:871: finishing write tid 3 to smithi17031214-47 2024-03-27T09:38:37.554 INFO:tasks.rados.rados.0.smithi170.stdout:871: finishing write tid 4 to smithi17031214-47 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: finishing write tid 5 to smithi17031214-47 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: oid 47 updating version 0 to 157 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: oid 47 updating version 157 to 159 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: oid 47 version 159 is already newer than 158 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: oid 47 version 159 is already newer than 156 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 159 (ObjNum 420 snap 99 seq_num 420) dirty exists 2024-03-27T09:38:37.556 INFO:tasks.rados.rados.0.smithi170.stdout:871: left oid 47 (ObjNum 420 snap 99 seq_num 420) 2024-03-27T09:38:37.561 INFO:tasks.rados.rados.0.smithi170.stdout:874: finishing write tid 1 to smithi17031214-50 2024-03-27T09:38:37.579 INFO:tasks.rados.rados.0.smithi170.stdout:874: finishing write tid 2 to smithi17031214-50 2024-03-27T09:38:37.597 INFO:tasks.rados.rados.0.smithi170.stdout:874: finishing write tid 3 to smithi17031214-50 2024-03-27T09:38:37.606 INFO:tasks.rados.rados.0.smithi170.stdout:874: finishing write tid 4 to smithi17031214-50 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: finishing write tid 5 to smithi17031214-50 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: oid 50 updating version 0 to 309 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: oid 50 updating version 309 to 310 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: oid 50 updating version 310 to 311 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: oid 50 updating version 311 to 312 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 312 (ObjNum 422 snap 99 seq_num 422) dirty exists 2024-03-27T09:38:37.608 INFO:tasks.rados.rados.0.smithi170.stdout:874: left oid 50 (ObjNum 422 snap 99 seq_num 422) 2024-03-27T09:38:38.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:38.125 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:38.122+0000 7f5239dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:38.128 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:38.127+0000 7f5239dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:38.152 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:38.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:867: done (8 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:868: done (7 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:869: done (6 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:870: done (5 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:871: done (4 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:872: done (3 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:873: done (2 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:874: done (1 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:875: done (0 left) 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:876: write (excl) oid 48 current snap is 100 2024-03-27T09:38:38.204 INFO:tasks.rados.rados.0.smithi170.stdout:876: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-03-27T09:38:38.211 INFO:tasks.rados.rados.0.smithi170.stdout:876: writing smithi17031214-48 from 709361 to 1339103 tid 1 2024-03-27T09:38:38.217 INFO:tasks.rados.rados.0.smithi170.stdout:876: writing smithi17031214-48 from 1836589 to 2376405 tid 2 2024-03-27T09:38:38.221 INFO:tasks.rados.rados.0.smithi170.stdout:876: writing smithi17031214-48 from 3033220 to 3109361 tid 3 2024-03-27T09:38:38.221 INFO:tasks.rados.rados.0.smithi170.stdout:877: delete oid 24 current snap is 100 2024-03-27T09:38:38.260 INFO:tasks.rados.rados.0.smithi170.stdout:877: done (1 left) 2024-03-27T09:38:38.260 INFO:tasks.rados.rados.0.smithi170.stdout:878: snap_remove snap 99 2024-03-27T09:38:39.208 INFO:tasks.rados.rados.0.smithi170.stdout:878: done (1 left) 2024-03-27T09:38:39.208 INFO:tasks.rados.rados.0.smithi170.stdout:879: write (excl) oid 42 current snap is 100 2024-03-27T09:38:39.208 INFO:tasks.rados.rados.0.smithi170.stdout:879: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-03-27T09:38:39.216 INFO:tasks.rados.rados.0.smithi170.stdout:879: writing smithi17031214-42 from 742975 to 1523215 tid 1 2024-03-27T09:38:39.221 INFO:tasks.rados.rados.0.smithi170.stdout:879: writing smithi17031214-42 from 1953553 to 2380685 tid 2 2024-03-27T09:38:39.225 INFO:tasks.rados.rados.0.smithi170.stdout:879: writing smithi17031214-42 from 2971713 to 3142975 tid 3 2024-03-27T09:38:39.225 INFO:tasks.rados.rados.0.smithi170.stdout:880: snap_remove snap 98 2024-03-27T09:38:39.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:39.874 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:39.877 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:40.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:40.212 INFO:tasks.rados.rados.0.smithi170.stdout:880: done (2 left) 2024-03-27T09:38:40.212 INFO:tasks.rados.rados.0.smithi170.stdout:881: write (excl) oid 22 current snap is 100 2024-03-27T09:38:40.212 INFO:tasks.rados.rados.0.smithi170.stdout:876: finishing write tid 1 to smithi17031214-48 2024-03-27T09:38:40.213 INFO:tasks.rados.rados.0.smithi170.stdout:881: seq_num 426 ranges {759782=571842,2012035=687143} 2024-03-27T09:38:40.220 INFO:tasks.rados.rados.0.smithi170.stdout:881: writing smithi17031214-22 from 759782 to 1331624 tid 1 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:881: writing smithi17031214-22 from 2012035 to 2699178 tid 2 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:882: delete oid 16 current snap is 100 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: finishing write tid 2 to smithi17031214-48 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: finishing write tid 3 to smithi17031214-48 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: finishing write tid 4 to smithi17031214-48 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: finishing write tid 5 to smithi17031214-48 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: oid 48 updating version 0 to 224 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: oid 48 version 224 is already newer than 222 2024-03-27T09:38:40.228 INFO:tasks.rados.rados.0.smithi170.stdout:876: oid 48 version 224 is already newer than 223 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:876: oid 48 updating version 224 to 225 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 225 (ObjNum 423 snap 100 seq_num 423) dirty exists 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:876: left oid 48 (ObjNum 423 snap 100 seq_num 423) 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: finishing write tid 1 to smithi17031214-42 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: finishing write tid 2 to smithi17031214-42 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: finishing write tid 3 to smithi17031214-42 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: finishing write tid 4 to smithi17031214-42 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: finishing write tid 5 to smithi17031214-42 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: oid 42 updating version 0 to 368 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: oid 42 updating version 368 to 371 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: oid 42 version 371 is already newer than 369 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: oid 42 version 371 is already newer than 371 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 371 (ObjNum 425 snap 100 seq_num 425) dirty exists 2024-03-27T09:38:40.229 INFO:tasks.rados.rados.0.smithi170.stdout:879: left oid 42 (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:38:40.338 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:40.336+0000 7f9673b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:40.342 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:40.341+0000 7f9673b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:40.343 INFO:tasks.rados.rados.0.smithi170.stdout:881: finishing write tid 1 to smithi17031214-22 2024-03-27T09:38:40.365 INFO:tasks.rados.rados.0.smithi170.stdout:876: done (3 left) 2024-03-27T09:38:40.365 INFO:tasks.rados.rados.0.smithi170.stdout:879: done (2 left) 2024-03-27T09:38:40.365 INFO:tasks.rados.rados.0.smithi170.stdout:882: done (1 left) 2024-03-27T09:38:40.366 INFO:tasks.rados.rados.0.smithi170.stdout:883: write (excl) oid 15 current snap is 100 2024-03-27T09:38:40.366 INFO:tasks.rados.rados.0.smithi170.stdout:883: seq_num 428 ranges {793396=722340,2128999=690812} 2024-03-27T09:38:40.369 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:40.374 INFO:tasks.rados.rados.0.smithi170.stdout:883: writing smithi17031214-15 from 793396 to 1515736 tid 1 2024-03-27T09:38:40.381 INFO:tasks.rados.rados.0.smithi170.stdout:883: writing smithi17031214-15 from 2128999 to 2819811 tid 2 2024-03-27T09:38:40.381 INFO:tasks.rados.rados.0.smithi170.stdout:884: write oid 33 current snap is 100 2024-03-27T09:38:40.381 INFO:tasks.rados.rados.0.smithi170.stdout:884: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-03-27T09:38:40.388 INFO:tasks.rados.rados.0.smithi170.stdout:884: writing smithi17031214-33 from 410203 to 1207792 tid 1 2024-03-27T09:38:40.393 INFO:tasks.rados.rados.0.smithi170.stdout:884: writing smithi17031214-33 from 1787481 to 2221951 tid 2 2024-03-27T09:38:40.399 INFO:tasks.rados.rados.0.smithi170.stdout:884: writing smithi17031214-33 from 2681405 to 3210203 tid 3 2024-03-27T09:38:40.399 INFO:tasks.rados.rados.0.smithi170.stdout:885: delete oid 46 current snap is 100 2024-03-27T09:38:40.399 INFO:tasks.rados.rados.0.smithi170.stdout:881: finishing write tid 2 to smithi17031214-22 2024-03-27T09:38:40.408 INFO:tasks.rados.rados.0.smithi170.stdout:883: finishing write tid 1 to smithi17031214-15 2024-03-27T09:38:40.411 INFO:tasks.rados.rados.0.smithi170.stdout:881: finishing write tid 3 to smithi17031214-22 2024-03-27T09:38:40.414 INFO:tasks.rados.rados.0.smithi170.stdout:881: finishing write tid 4 to smithi17031214-22 2024-03-27T09:38:40.414 INFO:tasks.rados.rados.0.smithi170.stdout:881: oid 22 updating version 0 to 176 2024-03-27T09:38:40.414 INFO:tasks.rados.rados.0.smithi170.stdout:881: oid 22 version 176 is already newer than 175 2024-03-27T09:38:40.415 INFO:tasks.rados.rados.0.smithi170.stdout:881: oid 22 updating version 176 to 177 2024-03-27T09:38:40.415 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 177 (ObjNum 426 snap 100 seq_num 426) dirty exists 2024-03-27T09:38:40.415 INFO:tasks.rados.rados.0.smithi170.stdout:881: left oid 22 (ObjNum 426 snap 100 seq_num 426) 2024-03-27T09:38:40.434 INFO:tasks.rados.rados.0.smithi170.stdout:884: finishing write tid 1 to smithi17031214-33 2024-03-27T09:38:40.435 INFO:tasks.rados.rados.0.smithi170.stdout:881: done (3 left) 2024-03-27T09:38:40.435 INFO:tasks.rados.rados.0.smithi170.stdout:885: done (2 left) 2024-03-27T09:38:40.436 INFO:tasks.rados.rados.0.smithi170.stdout:886: write oid 38 current snap is 100 2024-03-27T09:38:40.436 INFO:tasks.rados.rados.0.smithi170.stdout:886: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-03-27T09:38:40.441 INFO:tasks.rados.rados.0.smithi170.stdout:886: writing smithi17031214-38 from 443817 to 991904 tid 1 2024-03-27T09:38:40.448 INFO:tasks.rados.rados.0.smithi170.stdout:886: writing smithi17031214-38 from 1788092 to 2226231 tid 2 2024-03-27T09:38:40.454 INFO:tasks.rados.rados.0.smithi170.stdout:886: writing smithi17031214-38 from 3019898 to 3243817 tid 3 2024-03-27T09:38:40.454 INFO:tasks.rados.rados.0.smithi170.stdout:887: write (excl) oid 21 current snap is 100 2024-03-27T09:38:40.454 INFO:tasks.rados.rados.0.smithi170.stdout:883: finishing write tid 2 to smithi17031214-15 2024-03-27T09:38:40.454 INFO:tasks.rados.rados.0.smithi170.stdout:887: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-03-27T09:38:40.460 INFO:tasks.rados.rados.0.smithi170.stdout:887: writing smithi17031214-21 from 460624 to 1083960 tid 1 2024-03-27T09:38:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:40.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:40.468 INFO:tasks.rados.rados.0.smithi170.stdout:887: writing smithi17031214-21 from 1846574 to 2544724 tid 2 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:887: writing smithi17031214-21 from 3163674 to 3260624 tid 3 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:888: read oid 13 snap -1 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:888: expect deleted 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:889: read oid 36 snap -1 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:889: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:890: write oid 6 current snap is 100 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:884: finishing write tid 2 to smithi17031214-33 2024-03-27T09:38:40.473 INFO:tasks.rados.rados.0.smithi170.stdout:890: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-03-27T09:38:40.479 INFO:tasks.rados.rados.0.smithi170.stdout:890: writing smithi17031214-6 from 477431 to 1176016 tid 1 2024-03-27T09:38:40.485 INFO:tasks.rados.rados.0.smithi170.stdout:890: writing smithi17031214-6 from 1905056 to 2346864 tid 2 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:890: writing smithi17031214-6 from 3074744 to 3277431 tid 3 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:891: read oid 23 snap -1 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:891: expect deleted 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:892: snap_create 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:884: finishing write tid 3 to smithi17031214-33 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: finishing write tid 3 to smithi17031214-15 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:886: finishing write tid 1 to smithi17031214-38 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: finishing write tid 4 to smithi17031214-15 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: oid 15 updating version 0 to 146 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: oid 15 updating version 146 to 147 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: oid 15 updating version 147 to 148 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 148 (ObjNum 428 snap 100 seq_num 428) dirty exists 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:883: left oid 15 (ObjNum 428 snap 100 seq_num 428) 2024-03-27T09:38:40.492 INFO:tasks.rados.rados.0.smithi170.stdout:884: finishing write tid 4 to smithi17031214-33 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: finishing write tid 5 to smithi17031214-33 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: oid 33 updating version 0 to 230 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: oid 33 updating version 230 to 233 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: oid 33 version 233 is already newer than 232 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: oid 33 version 233 is already newer than 232 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 233 (ObjNum 429 snap 100 seq_num 429) dirty exists 2024-03-27T09:38:40.493 INFO:tasks.rados.rados.0.smithi170.stdout:884: left oid 33 (ObjNum 429 snap 100 seq_num 429) 2024-03-27T09:38:40.507 INFO:tasks.rados.rados.0.smithi170.stdout:886: finishing write tid 2 to smithi17031214-38 2024-03-27T09:38:40.512 INFO:tasks.rados.rados.0.smithi170.stdout:889: expect (ObjNum 226 snap 54 seq_num 226) 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:890: finishing write tid 1 to smithi17031214-6 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: finishing write tid 3 to smithi17031214-38 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: finishing write tid 4 to smithi17031214-38 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:890: finishing write tid 2 to smithi17031214-6 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: finishing write tid 5 to smithi17031214-38 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: oid 38 updating version 0 to 240 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: oid 38 updating version 240 to 242 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: oid 38 version 242 is already newer than 239 2024-03-27T09:38:40.777 INFO:tasks.rados.rados.0.smithi170.stdout:886: oid 38 version 242 is already newer than 242 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 242 (ObjNum 431 snap 100 seq_num 431) dirty exists 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:886: left oid 38 (ObjNum 431 snap 100 seq_num 431) 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:887: finishing write tid 1 to smithi17031214-21 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: finishing write tid 3 to smithi17031214-6 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: finishing write tid 4 to smithi17031214-6 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: finishing write tid 5 to smithi17031214-6 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: oid 6 updating version 0 to 179 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: oid 6 updating version 179 to 180 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: oid 6 version 180 is already newer than 178 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: oid 6 updating version 180 to 181 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 181 (ObjNum 433 snap 100 seq_num 433) dirty exists 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:890: left oid 6 (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:887: finishing write tid 2 to smithi17031214-21 2024-03-27T09:38:40.778 INFO:tasks.rados.rados.0.smithi170.stdout:887: finishing write tid 3 to smithi17031214-21 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: finishing write tid 4 to smithi17031214-21 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: finishing write tid 5 to smithi17031214-21 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: oid 21 updating version 0 to 246 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: oid 21 version 246 is already newer than 243 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: oid 21 version 246 is already newer than 245 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: oid 21 version 246 is already newer than 245 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 246 (ObjNum 432 snap 100 seq_num 432) dirty exists 2024-03-27T09:38:40.779 INFO:tasks.rados.rados.0.smithi170.stdout:887: left oid 21 (ObjNum 432 snap 100 seq_num 432) 2024-03-27T09:38:41.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~171s 2024-03-27T09:38:41.214 INFO:tasks.rados.rados.0.smithi170.stdout:883: done (8 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:884: done (7 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:886: done (6 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:887: done (5 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:888: done (4 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:889: done (3 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:890: done (2 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:891: done (1 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:892: done (0 left) 2024-03-27T09:38:41.215 INFO:tasks.rados.rados.0.smithi170.stdout:893: snap_create 2024-03-27T09:38:42.219 INFO:tasks.rados.rados.0.smithi170.stdout:893: done (0 left) 2024-03-27T09:38:42.219 INFO:tasks.rados.rados.0.smithi170.stdout:894: delete oid 13 current snap is 102 2024-03-27T09:38:42.284 INFO:tasks.rados.rados.0.smithi170.stdout:894: done (0 left) 2024-03-27T09:38:42.284 INFO:tasks.rados.rados.0.smithi170.stdout:895: delete oid 45 current snap is 102 2024-03-27T09:38:42.340 INFO:tasks.rados.rados.0.smithi170.stdout:895: done (0 left) 2024-03-27T09:38:42.340 INFO:tasks.rados.rados.0.smithi170.stdout:896: delete oid 48 current snap is 102 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:896: done (0 left) 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:897: read oid 11 snap -1 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:897: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:898: read oid 46 snap -1 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:898: expect deleted 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:899: read oid 16 snap -1 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:899: expect deleted 2024-03-27T09:38:42.367 INFO:tasks.rados.rados.0.smithi170.stdout:900: snap_create 2024-03-27T09:38:42.393 INFO:tasks.rados.rados.0.smithi170.stdout:897: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:42.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:42.574 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:42.571+0000 7f9931264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:42.577 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:42.577+0000 7f9931264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:42.601 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:42.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:42.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:43.222 INFO:tasks.rados.rados.0.smithi170.stdout:897: done (3 left) 2024-03-27T09:38:43.222 INFO:tasks.rados.rados.0.smithi170.stdout:898: done (2 left) 2024-03-27T09:38:43.222 INFO:tasks.rados.rados.0.smithi170.stdout:899: done (1 left) 2024-03-27T09:38:43.222 INFO:tasks.rados.rados.0.smithi170.stdout:900: done (0 left) 2024-03-27T09:38:43.222 INFO:tasks.rados.rados.0.smithi170.stdout:901: delete oid 49 current snap is 103 2024-03-27T09:38:43.266 INFO:tasks.rados.rados.0.smithi170.stdout:901: done (0 left) 2024-03-27T09:38:43.266 INFO:tasks.rados.rados.0.smithi170.stdout:902: write oid 8 current snap is 103 2024-03-27T09:38:43.266 INFO:tasks.rados.rados.0.smithi170.stdout:902: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-03-27T09:38:43.273 INFO:tasks.rados.rados.0.smithi170.stdout:902: writing smithi17031214-8 from 561466 to 1352649 tid 1 2024-03-27T09:38:43.280 INFO:tasks.rados.rados.0.smithi170.stdout:902: writing smithi17031214-8 from 2081113 to 2673917 tid 2 2024-03-27T09:38:43.284 INFO:tasks.rados.rados.0.smithi170.stdout:902: writing smithi17031214-8 from 3095506 to 3361466 tid 3 2024-03-27T09:38:43.284 INFO:tasks.rados.rados.0.smithi170.stdout:903: write (excl) oid 46 current snap is 103 2024-03-27T09:38:43.284 INFO:tasks.rados.rados.0.smithi170.stdout:903: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-03-27T09:38:43.290 INFO:tasks.rados.rados.0.smithi170.stdout:903: writing smithi17031214-46 from 578273 to 1044705 tid 1 2024-03-27T09:38:43.296 INFO:tasks.rados.rados.0.smithi170.stdout:903: writing smithi17031214-46 from 1739595 to 2192410 tid 2 2024-03-27T09:38:43.303 INFO:tasks.rados.rados.0.smithi170.stdout:903: writing smithi17031214-46 from 2722929 to 3378273 tid 3 2024-03-27T09:38:43.303 INFO:tasks.rados.rados.0.smithi170.stdout:904: snap_remove snap 102 2024-03-27T09:38:44.226 INFO:tasks.rados.rados.0.smithi170.stdout:904: done (2 left) 2024-03-27T09:38:44.226 INFO:tasks.rados.rados.0.smithi170.stdout:905: snap_remove snap 101 2024-03-27T09:38:44.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:44.768 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:44.766+0000 7f0de60ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:44.772 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:44.771+0000 7f0de60ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:44.796 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:44.848 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:44.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:44.978 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:44.981 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:45.227 INFO:tasks.rados.rados.0.smithi170.stdout:905: done (2 left) 2024-03-27T09:38:45.227 INFO:tasks.rados.rados.0.smithi170.stdout:906: read oid 7 snap -1 2024-03-27T09:38:45.227 INFO:tasks.rados.rados.0.smithi170.stdout:906: expect (ObjNum 383 snap 93 seq_num 383) 2024-03-27T09:38:45.227 INFO:tasks.rados.rados.0.smithi170.stdout:907: snap_remove snap 100 2024-03-27T09:38:45.972 INFO:tasks.rados.rados.0.smithi170.stdout:907: done (3 left) 2024-03-27T09:38:45.973 INFO:tasks.rados.rados.0.smithi170.stdout:908: write oid 28 current snap is 103 2024-03-27T09:38:45.973 INFO:tasks.rados.rados.0.smithi170.stdout:903: finishing write tid 1 to smithi17031214-46 2024-03-27T09:38:45.973 INFO:tasks.rados.rados.0.smithi170.stdout:908: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-03-27T09:38:45.978 INFO:tasks.rados.rados.0.smithi170.stdout:908: writing smithi17031214-28 from 595080 to 1136761 tid 1 2024-03-27T09:38:45.988 INFO:tasks.rados.rados.0.smithi170.stdout:908: writing smithi17031214-28 from 1798077 to 2394550 tid 2 2024-03-27T09:38:45.995 INFO:tasks.rados.rados.0.smithi170.stdout:908: writing smithi17031214-28 from 3150352 to 3395080 tid 3 2024-03-27T09:38:45.995 INFO:tasks.rados.rados.0.smithi170.stdout:909: read oid 23 snap -1 2024-03-27T09:38:45.995 INFO:tasks.rados.rados.0.smithi170.stdout:909: expect deleted 2024-03-27T09:38:45.995 INFO:tasks.rados.rados.0.smithi170.stdout:910: delete oid 34 current snap is 103 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: finishing write tid 1 to smithi17031214-8 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:903: finishing write tid 2 to smithi17031214-46 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: finishing write tid 2 to smithi17031214-8 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: finishing write tid 3 to smithi17031214-8 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:903: finishing write tid 3 to smithi17031214-46 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: finishing write tid 4 to smithi17031214-8 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:903: finishing write tid 4 to smithi17031214-46 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: finishing write tid 5 to smithi17031214-8 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: oid 8 updating version 0 to 317 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: oid 8 updating version 317 to 318 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: oid 8 version 318 is already newer than 315 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: oid 8 version 318 is already newer than 317 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 318 (ObjNum 438 snap 103 seq_num 438) dirty exists 2024-03-27T09:38:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:902: left oid 8 (ObjNum 438 snap 103 seq_num 438) 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: finishing write tid 5 to smithi17031214-46 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: oid 46 updating version 0 to 373 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: oid 46 updating version 373 to 374 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: oid 46 updating version 374 to 375 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: oid 46 updating version 375 to 376 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 376 (ObjNum 439 snap 103 seq_num 439) dirty exists 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:903: left oid 46 (ObjNum 439 snap 103 seq_num 439) 2024-03-27T09:38:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:906: expect (ObjNum 383 snap 93 seq_num 383) 2024-03-27T09:38:46.224 INFO:tasks.rados.rados.0.smithi170.stdout:908: finishing write tid 1 to smithi17031214-28 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:908: finishing write tid 2 to smithi17031214-28 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:902: done (5 left) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:903: done (4 left) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:906: done (3 left) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:909: done (2 left) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:910: done (1 left) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:911: read oid 10 snap -1 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:911: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:912: write oid 24 current snap is 103 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:908: finishing write tid 3 to smithi17031214-28 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:908: finishing write tid 4 to smithi17031214-28 2024-03-27T09:38:46.225 INFO:tasks.rados.rados.0.smithi170.stdout:908: finishing write tid 5 to smithi17031214-28 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:908: oid 28 updating version 0 to 327 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:908: oid 28 version 327 is already newer than 326 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:908: oid 28 version 327 is already newer than 325 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:908: oid 28 updating version 327 to 328 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 328 (ObjNum 440 snap 103 seq_num 440) dirty exists 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:908: left oid 28 (ObjNum 440 snap 103 seq_num 440) 2024-03-27T09:38:46.226 INFO:tasks.rados.rados.0.smithi170.stdout:912: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-03-27T09:38:46.229 INFO:tasks.rados.rados.0.smithi170.stdout:912: writing smithi17031214-24 from 628694 to 1037226 tid 1 2024-03-27T09:38:46.236 INFO:tasks.rados.rados.0.smithi170.stdout:912: writing smithi17031214-24 from 1515041 to 2115183 tid 2 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:912: writing smithi17031214-24 from 2805198 to 3357310 tid 3 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:908: done (2 left) 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:913: read oid 1 snap -1 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:913: expect deleted 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:914: delete oid 31 current snap is 103 2024-03-27T09:38:46.244 INFO:tasks.rados.rados.0.smithi170.stdout:911: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: finishing write tid 1 to smithi17031214-24 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: finishing write tid 2 to smithi17031214-24 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: finishing write tid 3 to smithi17031214-24 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: finishing write tid 4 to smithi17031214-24 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: finishing write tid 5 to smithi17031214-24 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: oid 24 updating version 0 to 151 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: oid 24 updating version 151 to 152 2024-03-27T09:38:46.388 INFO:tasks.rados.rados.0.smithi170.stdout:912: oid 24 version 152 is already newer than 149 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:912: oid 24 version 152 is already newer than 150 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 152 (ObjNum 442 snap 103 seq_num 442) dirty exists 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:912: left oid 24 (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:911: done (3 left) 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:912: done (2 left) 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:913: done (1 left) 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:914: done (0 left) 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:915: read oid 48 snap -1 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:915: expect deleted 2024-03-27T09:38:46.389 INFO:tasks.rados.rados.0.smithi170.stdout:916: delete oid 47 current snap is 103 2024-03-27T09:38:46.405 INFO:tasks.rados.rados.0.smithi170.stdout:915: done (1 left) 2024-03-27T09:38:46.405 INFO:tasks.rados.rados.0.smithi170.stdout:916: done (0 left) 2024-03-27T09:38:46.405 INFO:tasks.rados.rados.0.smithi170.stdout:917: write (excl) oid 30 current snap is 103 2024-03-27T09:38:46.405 INFO:tasks.rados.rados.0.smithi170.stdout:917: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-03-27T09:38:46.412 INFO:tasks.rados.rados.0.smithi170.stdout:917: writing smithi17031214-30 from 679115 to 1313394 tid 1 2024-03-27T09:38:46.419 INFO:tasks.rados.rados.0.smithi170.stdout:917: writing smithi17031214-30 from 2090487 to 2554309 tid 2 2024-03-27T09:38:46.424 INFO:tasks.rados.rados.0.smithi170.stdout:917: writing smithi17031214-30 from 3003820 to 3479115 tid 3 2024-03-27T09:38:46.424 INFO:tasks.rados.rados.0.smithi170.stdout:918: write (excl) oid 8 current snap is 103 2024-03-27T09:38:46.424 INFO:tasks.rados.rados.0.smithi170.stdout:918: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-03-27T09:38:46.432 INFO:tasks.rados.rados.0.smithi170.stdout:918: writing smithi17031214-8 from 695922 to 1405450 tid 1 2024-03-27T09:38:46.439 INFO:tasks.rados.rados.0.smithi170.stdout:918: writing smithi17031214-8 from 2032616 to 2640096 tid 2 2024-03-27T09:38:46.444 INFO:tasks.rados.rados.0.smithi170.stdout:918: writing smithi17031214-8 from 3198537 to 3495922 tid 3 2024-03-27T09:38:46.444 INFO:tasks.rados.rados.0.smithi170.stdout:919: snap_create 2024-03-27T09:38:46.444 INFO:tasks.rados.rados.0.smithi170.stdout:917: finishing write tid 1 to smithi17031214-30 2024-03-27T09:38:46.461 INFO:tasks.rados.rados.0.smithi170.stdout:918: finishing write tid 1 to smithi17031214-8 2024-03-27T09:38:46.464 INFO:tasks.rados.rados.0.smithi170.stdout:917: finishing write tid 2 to smithi17031214-30 2024-03-27T09:38:46.481 INFO:tasks.rados.rados.0.smithi170.stdout:918: finishing write tid 2 to smithi17031214-8 2024-03-27T09:38:46.482 INFO:tasks.rados.rados.0.smithi170.stdout:917: finishing write tid 3 to smithi17031214-30 2024-03-27T09:38:46.498 INFO:tasks.rados.rados.0.smithi170.stdout:917: finishing write tid 4 to smithi17031214-30 2024-03-27T09:38:46.499 INFO:tasks.rados.rados.0.smithi170.stdout:918: finishing write tid 3 to smithi17031214-8 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: finishing write tid 5 to smithi17031214-30 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: oid 30 updating version 0 to 203 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: oid 30 version 203 is already newer than 201 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: oid 30 version 203 is already newer than 200 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: oid 30 version 203 is already newer than 202 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 203 (ObjNum 445 snap 103 seq_num 445) dirty exists 2024-03-27T09:38:46.500 INFO:tasks.rados.rados.0.smithi170.stdout:917: left oid 30 (ObjNum 445 snap 103 seq_num 445) 2024-03-27T09:38:46.505 INFO:tasks.rados.rados.0.smithi170.stdout:918: finishing write tid 4 to smithi17031214-8 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: finishing write tid 5 to smithi17031214-8 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: oid 8 updating version 0 to 322 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: oid 8 version 322 is already newer than 319 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: oid 8 version 322 is already newer than 322 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: oid 8 version 322 is already newer than 320 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 322 (ObjNum 446 snap 103 seq_num 446) dirty exists 2024-03-27T09:38:46.507 INFO:tasks.rados.rados.0.smithi170.stdout:918: left oid 8 (ObjNum 446 snap 103 seq_num 446) 2024-03-27T09:38:46.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~176s 2024-03-27T09:38:46.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:46.960 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:46.958+0000 7fd704c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:46.964 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:46.964+0000 7fd704c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:46.988 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:47.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:917: done (2 left) 2024-03-27T09:38:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:918: done (1 left) 2024-03-27T09:38:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:919: done (0 left) 2024-03-27T09:38:47.237 INFO:tasks.rados.rados.0.smithi170.stdout:920: read oid 48 snap -1 2024-03-27T09:38:47.237 INFO:tasks.rados.rados.0.smithi170.stdout:920: expect deleted 2024-03-27T09:38:47.237 INFO:tasks.rados.rados.0.smithi170.stdout:921: delete oid 38 current snap is 104 2024-03-27T09:38:47.301 INFO:tasks.rados.rados.0.smithi170.stdout:920: done (1 left) 2024-03-27T09:38:47.301 INFO:tasks.rados.rados.0.smithi170.stdout:921: done (0 left) 2024-03-27T09:38:47.301 INFO:tasks.rados.rados.0.smithi170.stdout:922: write oid 33 current snap is 104 2024-03-27T09:38:47.301 INFO:tasks.rados.rados.0.smithi170.stdout:922: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-03-27T09:38:47.308 INFO:tasks.rados.rados.0.smithi170.stdout:922: writing smithi17031214-33 from 729536 to 1189562 tid 1 2024-03-27T09:38:47.314 INFO:tasks.rados.rados.0.smithi170.stdout:922: writing smithi17031214-33 from 1749580 to 2360729 tid 2 2024-03-27T09:38:47.320 INFO:tasks.rados.rados.0.smithi170.stdout:922: writing smithi17031214-33 from 2853383 to 3305833 tid 3 2024-03-27T09:38:47.320 INFO:tasks.rados.rados.0.smithi170.stdout:923: write oid 26 current snap is 104 2024-03-27T09:38:47.320 INFO:tasks.rados.rados.0.smithi170.stdout:923: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-03-27T09:38:47.327 INFO:tasks.rados.rados.0.smithi170.stdout:923: writing smithi17031214-26 from 746343 to 1397971 tid 1 2024-03-27T09:38:47.335 INFO:tasks.rados.rados.0.smithi170.stdout:923: writing smithi17031214-26 from 1924415 to 2679222 tid 2 2024-03-27T09:38:47.339 INFO:tasks.rados.rados.0.smithi170.stdout:923: writing smithi17031214-26 from 3397159 to 3546343 tid 3 2024-03-27T09:38:47.340 INFO:tasks.rados.rados.0.smithi170.stdout:924: write (excl) oid 27 current snap is 104 2024-03-27T09:38:47.340 INFO:tasks.rados.rados.0.smithi170.stdout:924: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-03-27T09:38:47.349 INFO:tasks.rados.rados.0.smithi170.stdout:924: writing smithi17031214-27 from 763150 to 1490027 tid 1 2024-03-27T09:38:47.355 INFO:tasks.rados.rados.0.smithi170.stdout:924: writing smithi17031214-27 from 1982897 to 2597715 tid 2 2024-03-27T09:38:47.360 INFO:tasks.rados.rados.0.smithi170.stdout:924: writing smithi17031214-27 from 3024582 to 3499576 tid 3 2024-03-27T09:38:47.360 INFO:tasks.rados.rados.0.smithi170.stdout:925: write (excl) oid 19 current snap is 104 2024-03-27T09:38:47.360 INFO:tasks.rados.rados.0.smithi170.stdout:925: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-03-27T09:38:47.367 INFO:tasks.rados.rados.0.smithi170.stdout:925: writing smithi17031214-19 from 779957 to 1182083 tid 1 2024-03-27T09:38:47.376 INFO:tasks.rados.rados.0.smithi170.stdout:925: writing smithi17031214-19 from 1925026 to 2683502 tid 2 2024-03-27T09:38:47.381 INFO:tasks.rados.rados.0.smithi170.stdout:925: writing smithi17031214-19 from 3335652 to 3579957 tid 3 2024-03-27T09:38:47.381 INFO:tasks.rados.rados.0.smithi170.stdout:926: delete oid 21 current snap is 104 2024-03-27T09:38:47.381 INFO:tasks.rados.rados.0.smithi170.stdout:922: finishing write tid 1 to smithi17031214-33 2024-03-27T09:38:47.382 INFO:tasks.rados.rados.0.smithi170.stdout:922: finishing write tid 2 to smithi17031214-33 2024-03-27T09:38:47.382 INFO:tasks.rados.rados.0.smithi170.stdout:922: finishing write tid 3 to smithi17031214-33 2024-03-27T09:38:47.387 INFO:tasks.rados.rados.0.smithi170.stdout:922: finishing write tid 4 to smithi17031214-33 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: finishing write tid 5 to smithi17031214-33 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: oid 33 updating version 0 to 242 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: oid 33 updating version 242 to 245 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: oid 33 version 245 is already newer than 243 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: oid 33 version 245 is already newer than 245 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 245 (ObjNum 448 snap 104 seq_num 448) dirty exists 2024-03-27T09:38:47.389 INFO:tasks.rados.rados.0.smithi170.stdout:922: left oid 33 (ObjNum 448 snap 104 seq_num 448) 2024-03-27T09:38:47.394 INFO:tasks.rados.rados.0.smithi170.stdout:923: finishing write tid 1 to smithi17031214-26 2024-03-27T09:38:47.411 INFO:tasks.rados.rados.0.smithi170.stdout:924: finishing write tid 1 to smithi17031214-27 2024-03-27T09:38:47.426 INFO:tasks.rados.rados.0.smithi170.stdout:923: finishing write tid 2 to smithi17031214-26 2024-03-27T09:38:47.438 INFO:tasks.rados.rados.0.smithi170.stdout:924: finishing write tid 2 to smithi17031214-27 2024-03-27T09:38:47.452 INFO:tasks.rados.rados.0.smithi170.stdout:923: finishing write tid 3 to smithi17031214-26 2024-03-27T09:38:47.456 INFO:tasks.rados.rados.0.smithi170.stdout:924: finishing write tid 3 to smithi17031214-27 2024-03-27T09:38:47.457 INFO:tasks.rados.rados.0.smithi170.stdout:923: finishing write tid 4 to smithi17031214-26 2024-03-27T09:38:47.462 INFO:tasks.rados.rados.0.smithi170.stdout:924: finishing write tid 4 to smithi17031214-27 2024-03-27T09:38:47.464 INFO:tasks.rados.rados.0.smithi170.stdout:923: finishing write tid 5 to smithi17031214-26 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:923: oid 26 updating version 0 to 250 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:923: oid 26 updating version 250 to 253 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:923: oid 26 version 253 is already newer than 251 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:923: oid 26 version 253 is already newer than 253 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 253 (ObjNum 449 snap 104 seq_num 449) dirty exists 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:923: left oid 26 (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: finishing write tid 5 to smithi17031214-27 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: oid 27 updating version 0 to 323 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: oid 27 version 323 is already newer than 321 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: oid 27 version 323 is already newer than 323 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: oid 27 version 323 is already newer than 320 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 323 (ObjNum 450 snap 104 seq_num 450) dirty exists 2024-03-27T09:38:47.465 INFO:tasks.rados.rados.0.smithi170.stdout:924: left oid 27 (ObjNum 450 snap 104 seq_num 450) 2024-03-27T09:38:47.480 INFO:tasks.rados.rados.0.smithi170.stdout:925: finishing write tid 1 to smithi17031214-19 2024-03-27T09:38:47.499 INFO:tasks.rados.rados.0.smithi170.stdout:925: finishing write tid 2 to smithi17031214-19 2024-03-27T09:38:47.510 INFO:tasks.rados.rados.0.smithi170.stdout:925: finishing write tid 3 to smithi17031214-19 2024-03-27T09:38:47.523 INFO:tasks.rados.rados.0.smithi170.stdout:925: finishing write tid 4 to smithi17031214-19 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: finishing write tid 5 to smithi17031214-19 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: oid 19 updating version 0 to 258 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: oid 19 version 258 is already newer than 257 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: oid 19 version 258 is already newer than 255 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: oid 19 version 258 is already newer than 256 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 258 (ObjNum 451 snap 104 seq_num 451) dirty exists 2024-03-27T09:38:47.527 INFO:tasks.rados.rados.0.smithi170.stdout:925: left oid 19 (ObjNum 451 snap 104 seq_num 451) 2024-03-27T09:38:47.544 INFO:tasks.rados.rados.0.smithi170.stdout:922: done (4 left) 2024-03-27T09:38:47.544 INFO:tasks.rados.rados.0.smithi170.stdout:923: done (3 left) 2024-03-27T09:38:47.545 INFO:tasks.rados.rados.0.smithi170.stdout:924: done (2 left) 2024-03-27T09:38:47.545 INFO:tasks.rados.rados.0.smithi170.stdout:925: done (1 left) 2024-03-27T09:38:47.545 INFO:tasks.rados.rados.0.smithi170.stdout:926: done (0 left) 2024-03-27T09:38:47.545 INFO:tasks.rados.rados.0.smithi170.stdout:927: delete oid 43 current snap is 104 2024-03-27T09:38:47.572 INFO:tasks.rados.rados.0.smithi170.stdout:927: done (0 left) 2024-03-27T09:38:47.572 INFO:tasks.rados.rados.0.smithi170.stdout:928: snap_create 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:928: done (0 left) 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:929: read oid 42 snap -1 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:929: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:930: read oid 13 snap 103 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:930: expect deleted 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:931: read oid 41 snap 104 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:931: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:932: write (excl) oid 48 current snap is 105 2024-03-27T09:38:48.241 INFO:tasks.rados.rados.0.smithi170.stdout:932: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-03-27T09:38:48.248 INFO:tasks.rados.rados.0.smithi170.stdout:932: writing smithi17031214-48 from 430378 to 1058251 tid 1 2024-03-27T09:38:48.257 INFO:tasks.rados.rados.0.smithi170.stdout:932: writing smithi17031214-48 from 1700472 to 2322628 tid 2 2024-03-27T09:38:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:932: writing smithi17031214-48 from 3017921 to 3630378 tid 3 2024-03-27T09:38:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:933: write (excl) oid 8 current snap is 105 2024-03-27T09:38:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:933: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-03-27T09:38:48.273 INFO:tasks.rados.rados.0.smithi170.stdout:933: writing smithi17031214-8 from 447185 to 1150307 tid 1 2024-03-27T09:38:48.284 INFO:tasks.rados.rados.0.smithi170.stdout:933: writing smithi17031214-8 from 1642601 to 2408415 tid 2 2024-03-27T09:38:48.291 INFO:tasks.rados.rados.0.smithi170.stdout:933: writing smithi17031214-8 from 2928991 to 3376698 tid 3 2024-03-27T09:38:48.291 INFO:tasks.rados.rados.0.smithi170.stdout:934: snap_create 2024-03-27T09:38:48.324 INFO:tasks.rados.rados.0.smithi170.stdout:933: finishing write tid 1 to smithi17031214-8 2024-03-27T09:38:48.326 INFO:tasks.rados.rados.0.smithi170.stdout:929: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:38:48.451 INFO:tasks.rados.rados.0.smithi170.stdout:931: expect (ObjNum 241 snap 64 seq_num 241) 2024-03-27T09:38:48.567 INFO:tasks.rados.rados.0.smithi170.stdout:932: finishing write tid 1 to smithi17031214-48 2024-03-27T09:38:48.567 INFO:tasks.rados.rados.0.smithi170.stdout:933: finishing write tid 2 to smithi17031214-8 2024-03-27T09:38:48.567 INFO:tasks.rados.rados.0.smithi170.stdout:932: finishing write tid 2 to smithi17031214-48 2024-03-27T09:38:48.567 INFO:tasks.rados.rados.0.smithi170.stdout:933: finishing write tid 3 to smithi17031214-8 2024-03-27T09:38:48.567 INFO:tasks.rados.rados.0.smithi170.stdout:933: finishing write tid 4 to smithi17031214-8 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: finishing write tid 5 to smithi17031214-8 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: oid 8 updating version 0 to 327 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: oid 8 version 327 is already newer than 324 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: oid 8 version 327 is already newer than 325 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: oid 8 version 327 is already newer than 327 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 327 (ObjNum 455 snap 105 seq_num 455) dirty exists 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:933: left oid 8 (ObjNum 455 snap 105 seq_num 455) 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: finishing write tid 3 to smithi17031214-48 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: finishing write tid 4 to smithi17031214-48 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: finishing write tid 5 to smithi17031214-48 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: oid 48 updating version 0 to 247 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: oid 48 updating version 247 to 248 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: oid 48 version 248 is already newer than 246 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:932: oid 48 updating version 248 to 249 2024-03-27T09:38:48.568 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 249 (ObjNum 454 snap 105 seq_num 454) dirty exists 2024-03-27T09:38:48.569 INFO:tasks.rados.rados.0.smithi170.stdout:932: left oid 48 (ObjNum 454 snap 105 seq_num 454) 2024-03-27T09:38:49.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:49.144 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:49.142+0000 7fa29626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:49.148 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:49.147+0000 7fa29626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:49.173 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:49.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:929: done (5 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:930: done (4 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:931: done (3 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:932: done (2 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:933: done (1 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:934: done (0 left) 2024-03-27T09:38:49.243 INFO:tasks.rados.rados.0.smithi170.stdout:935: snap_create 2024-03-27T09:38:49.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:50.086 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:50.246 INFO:tasks.rados.rados.0.smithi170.stdout:935: done (0 left) 2024-03-27T09:38:50.246 INFO:tasks.rados.rados.0.smithi170.stdout:936: write (excl) oid 44 current snap is 107 2024-03-27T09:38:50.246 INFO:tasks.rados.rados.0.smithi170.stdout:936: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-03-27T09:38:50.254 INFO:tasks.rados.rados.0.smithi170.stdout:936: writing smithi17031214-44 from 463992 to 1242363 tid 1 2024-03-27T09:38:50.258 INFO:tasks.rados.rados.0.smithi170.stdout:936: writing smithi17031214-44 from 1701083 to 2326908 tid 2 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:936: writing smithi17031214-44 from 2956414 to 3615393 tid 3 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:937: read oid 11 snap 105 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:937: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:938: read oid 48 snap -1 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:938: expect (ObjNum 454 snap 105 seq_num 454) 2024-03-27T09:38:50.266 INFO:tasks.rados.rados.0.smithi170.stdout:939: snap_remove snap 103 2024-03-27T09:38:51.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:51.249 INFO:tasks.rados.rados.0.smithi170.stdout:939: done (3 left) 2024-03-27T09:38:51.250 INFO:tasks.rados.rados.0.smithi170.stdout:940: snap_remove snap 106 2024-03-27T09:38:51.351 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:51.349+0000 7f64e4f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:51.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:51.355+0000 7f64e4f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:51.386 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:51.439 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:52.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2024-03-27T09:38:52.253 INFO:tasks.rados.rados.0.smithi170.stdout:940: done (3 left) 2024-03-27T09:38:52.253 INFO:tasks.rados.rados.0.smithi170.stdout:941: write oid 19 current snap is 107 2024-03-27T09:38:52.253 INFO:tasks.rados.rados.0.smithi170.stdout:941: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-03-27T09:38:52.259 INFO:tasks.rados.rados.0.smithi170.stdout:941: writing smithi17031214-19 from 480799 to 1050772 tid 1 2024-03-27T09:38:52.265 INFO:tasks.rados.rados.0.smithi170.stdout:941: writing smithi17031214-19 from 1475918 to 2245401 tid 2 2024-03-27T09:38:52.271 INFO:tasks.rados.rados.0.smithi170.stdout:941: writing smithi17031214-19 from 2700190 to 3286794 tid 3 2024-03-27T09:38:52.271 INFO:tasks.rados.rados.0.smithi170.stdout:942: snap_remove snap 104 2024-03-27T09:38:53.257 INFO:tasks.rados.rados.0.smithi170.stdout:942: done (4 left) 2024-03-27T09:38:53.257 INFO:tasks.rados.rados.0.smithi170.stdout:943: snap_create 2024-03-27T09:38:53.257 INFO:tasks.rados.rados.0.smithi170.stdout:937: expect (ObjNum 388 snap 95 seq_num 388) 2024-03-27T09:38:53.354 INFO:tasks.rados.rados.0.smithi170.stdout:936: finishing write tid 1 to smithi17031214-44 2024-03-27T09:38:53.355 INFO:tasks.rados.rados.0.smithi170.stdout:938: expect (ObjNum 454 snap 105 seq_num 454) 2024-03-27T09:38:53.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: finishing write tid 2 to smithi17031214-44 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: finishing write tid 3 to smithi17031214-44 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: finishing write tid 4 to smithi17031214-44 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: finishing write tid 5 to smithi17031214-44 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: oid 44 updating version 0 to 331 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: oid 44 version 331 is already newer than 331 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: oid 44 version 331 is already newer than 329 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:936: oid 44 version 331 is already newer than 328 2024-03-27T09:38:53.492 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 331 (ObjNum 456 snap 107 seq_num 456) dirty exists 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:936: left oid 44 (ObjNum 456 snap 107 seq_num 456) 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: finishing write tid 1 to smithi17031214-19 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: finishing write tid 2 to smithi17031214-19 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: finishing write tid 3 to smithi17031214-19 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: finishing write tid 4 to smithi17031214-19 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: finishing write tid 5 to smithi17031214-19 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: oid 19 updating version 0 to 272 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: oid 19 updating version 272 to 273 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: oid 19 version 273 is already newer than 270 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: oid 19 version 273 is already newer than 272 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 273 (ObjNum 457 snap 107 seq_num 457) dirty exists 2024-03-27T09:38:53.493 INFO:tasks.rados.rados.0.smithi170.stdout:941: left oid 19 (ObjNum 457 snap 107 seq_num 457) 2024-03-27T09:38:53.548 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:53.546+0000 7f044b110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:53.552 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:53.551+0000 7f044b110640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:53.576 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:53.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:936: done (4 left) 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:937: done (3 left) 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:938: done (2 left) 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:941: done (1 left) 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:943: done (0 left) 2024-03-27T09:38:54.261 INFO:tasks.rados.rados.0.smithi170.stdout:944: snap_remove snap 105 2024-03-27T09:38:55.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:38:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:38:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:38:55.202 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:38:55.261 INFO:tasks.rados.rados.0.smithi170.stdout:944: done (0 left) 2024-03-27T09:38:55.262 INFO:tasks.rados.rados.0.smithi170.stdout:945: delete oid 19 current snap is 108 2024-03-27T09:38:55.407 INFO:tasks.rados.rados.0.smithi170.stdout:945: done (0 left) 2024-03-27T09:38:55.407 INFO:tasks.rados.rados.0.smithi170.stdout:946: write oid 48 current snap is 108 2024-03-27T09:38:55.407 INFO:tasks.rados.rados.0.smithi170.stdout:946: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-03-27T09:38:55.413 INFO:tasks.rados.rados.0.smithi170.stdout:946: writing smithi17031214-48 from 514413 to 1234884 tid 1 2024-03-27T09:38:55.421 INFO:tasks.rados.rados.0.smithi170.stdout:946: writing smithi17031214-48 from 1876529 to 2649681 tid 2 2024-03-27T09:38:55.428 INFO:tasks.rados.rados.0.smithi170.stdout:946: writing smithi17031214-48 from 3438683 to 3714413 tid 3 2024-03-27T09:38:55.429 INFO:tasks.rados.rados.0.smithi170.stdout:947: delete oid 50 current snap is 108 2024-03-27T09:38:55.468 INFO:tasks.rados.rados.0.smithi170.stdout:947: done (1 left) 2024-03-27T09:38:55.468 INFO:tasks.rados.rados.0.smithi170.stdout:948: snap_remove snap 107 2024-03-27T09:38:55.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:38:55.736 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:55.734+0000 7ff873a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:55.740 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:55.739+0000 7ff873a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:55.764 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:55.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:55.974 INFO:tasks.rados.rados.0.smithi170.stdout:948: done (1 left) 2024-03-27T09:38:55.974 INFO:tasks.rados.rados.0.smithi170.stdout:949: write oid 47 current snap is 108 2024-03-27T09:38:55.974 INFO:tasks.rados.rados.0.smithi170.stdout:946: finishing write tid 1 to smithi17031214-48 2024-03-27T09:38:55.974 INFO:tasks.rados.rados.0.smithi170.stdout:949: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-03-27T09:38:55.979 INFO:tasks.rados.rados.0.smithi170.stdout:949: writing smithi17031214-47 from 548027 to 1018996 tid 1 2024-03-27T09:38:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:949: writing smithi17031214-47 from 1593493 to 2370314 tid 2 2024-03-27T09:38:55.993 INFO:tasks.rados.rados.0.smithi170.stdout:949: writing smithi17031214-47 from 3093529 to 3725221 tid 3 2024-03-27T09:38:55.994 INFO:tasks.rados.rados.0.smithi170.stdout:950: write (excl) oid 38 current snap is 108 2024-03-27T09:38:55.994 INFO:tasks.rados.rados.0.smithi170.stdout:950: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-03-27T09:38:55.999 INFO:tasks.rados.rados.0.smithi170.stdout:950: writing smithi17031214-38 from 564834 to 1111052 tid 1 2024-03-27T09:38:56.005 INFO:tasks.rados.rados.0.smithi170.stdout:950: writing smithi17031214-38 from 1651975 to 2172454 tid 2 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:950: writing smithi17031214-38 from 2720952 to 3280269 tid 3 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:951: read oid 31 snap -1 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:951: expect deleted 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:952: read oid 4 snap -1 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:952: expect (ObjNum 419 snap 99 seq_num 419) 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:953: read oid 1 snap -1 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:953: expect deleted 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:954: delete oid 11 current snap is 108 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:946: finishing write tid 2 to smithi17031214-48 2024-03-27T09:38:56.013 INFO:tasks.rados.rados.0.smithi170.stdout:946: finishing write tid 3 to smithi17031214-48 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: finishing write tid 4 to smithi17031214-48 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: finishing write tid 5 to smithi17031214-48 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: oid 48 updating version 0 to 255 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: oid 48 version 255 is already newer than 253 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: oid 48 updating version 255 to 256 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: oid 48 version 256 is already newer than 255 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 256 (ObjNum 459 snap 108 seq_num 459) dirty exists 2024-03-27T09:38:56.014 INFO:tasks.rados.rados.0.smithi170.stdout:946: left oid 48 (ObjNum 459 snap 108 seq_num 459) 2024-03-27T09:38:56.070 INFO:tasks.rados.rados.0.smithi170.stdout:946: done (6 left) 2024-03-27T09:38:56.070 INFO:tasks.rados.rados.0.smithi170.stdout:951: done (5 left) 2024-03-27T09:38:56.070 INFO:tasks.rados.rados.0.smithi170.stdout:953: done (4 left) 2024-03-27T09:38:56.070 INFO:tasks.rados.rados.0.smithi170.stdout:954: done (3 left) 2024-03-27T09:38:56.070 INFO:tasks.rados.rados.0.smithi170.stdout:955: snap_create 2024-03-27T09:38:56.074 INFO:tasks.rados.rados.0.smithi170.stdout:949: finishing write tid 1 to smithi17031214-47 2024-03-27T09:38:56.090 INFO:tasks.rados.rados.0.smithi170.stdout:950: finishing write tid 1 to smithi17031214-38 2024-03-27T09:38:56.116 INFO:tasks.rados.rados.0.smithi170.stdout:952: expect (ObjNum 419 snap 99 seq_num 419) 2024-03-27T09:38:56.274 INFO:tasks.rados.rados.0.smithi170.stdout:949: finishing write tid 2 to smithi17031214-47 2024-03-27T09:38:56.274 INFO:tasks.rados.rados.0.smithi170.stdout:950: finishing write tid 2 to smithi17031214-38 2024-03-27T09:38:56.274 INFO:tasks.rados.rados.0.smithi170.stdout:949: finishing write tid 3 to smithi17031214-47 2024-03-27T09:38:56.274 INFO:tasks.rados.rados.0.smithi170.stdout:949: finishing write tid 4 to smithi17031214-47 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: finishing write tid 3 to smithi17031214-38 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: finishing write tid 5 to smithi17031214-47 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: oid 47 updating version 0 to 164 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: oid 47 version 164 is already newer than 163 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: oid 47 version 164 is already newer than 163 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: oid 47 version 164 is already newer than 161 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 164 (ObjNum 461 snap 108 seq_num 461) dirty exists 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:949: left oid 47 (ObjNum 461 snap 108 seq_num 461) 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: finishing write tid 4 to smithi17031214-38 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: finishing write tid 5 to smithi17031214-38 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: oid 38 updating version 0 to 278 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: oid 38 updating version 278 to 281 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: oid 38 version 281 is already newer than 281 2024-03-27T09:38:56.275 INFO:tasks.rados.rados.0.smithi170.stdout:950: oid 38 version 281 is already newer than 279 2024-03-27T09:38:56.276 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 281 (ObjNum 462 snap 108 seq_num 462) dirty exists 2024-03-27T09:38:56.276 INFO:tasks.rados.rados.0.smithi170.stdout:950: left oid 38 (ObjNum 462 snap 108 seq_num 462) 2024-03-27T09:38:56.975 INFO:tasks.rados.rados.0.smithi170.stdout:949: done (3 left) 2024-03-27T09:38:56.975 INFO:tasks.rados.rados.0.smithi170.stdout:950: done (2 left) 2024-03-27T09:38:56.975 INFO:tasks.rados.rados.0.smithi170.stdout:952: done (1 left) 2024-03-27T09:38:56.975 INFO:tasks.rados.rados.0.smithi170.stdout:955: done (0 left) 2024-03-27T09:38:56.975 INFO:tasks.rados.rados.0.smithi170.stdout:956: snap_create 2024-03-27T09:38:57.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2024-03-27T09:38:57.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:38:57.930 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:57.928+0000 7f2a5e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:57.934 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:38:57.933+0000 7f2a5e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:38:57.959 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:38:57.975 INFO:tasks.rados.rados.0.smithi170.stdout:956: done (0 left) 2024-03-27T09:38:57.975 INFO:tasks.rados.rados.0.smithi170.stdout:957: write (excl) oid 44 current snap is 110 2024-03-27T09:38:57.975 INFO:tasks.rados.rados.0.smithi170.stdout:957: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-03-27T09:38:57.981 INFO:tasks.rados.rados.0.smithi170.stdout:957: writing smithi17031214-44 from 598448 to 1011517 tid 1 2024-03-27T09:38:57.988 INFO:tasks.rados.rados.0.smithi170.stdout:957: writing smithi17031214-44 from 1768939 to 2293087 tid 2 2024-03-27T09:38:57.994 INFO:tasks.rados.rados.0.smithi170.stdout:957: writing smithi17031214-44 from 2775798 to 3357659 tid 3 2024-03-27T09:38:57.996 INFO:tasks.rados.rados.0.smithi170.stdout:957: writing smithi17031214-44 from 3781488 to 3798448 tid 4 2024-03-27T09:38:57.996 INFO:tasks.rados.rados.0.smithi170.stdout:958: delete oid 38 current snap is 110 2024-03-27T09:38:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:38:58.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:38:58.074 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 1 to smithi17031214-44 2024-03-27T09:38:58.084 INFO:tasks.rados.rados.0.smithi170.stdout:958: done (1 left) 2024-03-27T09:38:58.084 INFO:tasks.rados.rados.0.smithi170.stdout:959: write (excl) oid 13 current snap is 110 2024-03-27T09:38:58.084 INFO:tasks.rados.rados.0.smithi170.stdout:959: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-03-27T09:38:58.091 INFO:tasks.rados.rados.0.smithi170.stdout:959: writing smithi17031214-13 from 632062 to 1195629 tid 1 2024-03-27T09:38:58.098 INFO:tasks.rados.rados.0.smithi170.stdout:959: writing smithi17031214-13 from 1885903 to 2413720 tid 2 2024-03-27T09:38:58.105 INFO:tasks.rados.rados.0.smithi170.stdout:959: writing smithi17031214-13 from 2830644 to 3551402 tid 3 2024-03-27T09:38:58.106 INFO:tasks.rados.rados.0.smithi170.stdout:960: delete oid 17 current snap is 110 2024-03-27T09:38:58.106 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 2 to smithi17031214-44 2024-03-27T09:38:58.114 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 3 to smithi17031214-44 2024-03-27T09:38:58.130 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 4 to smithi17031214-44 2024-03-27T09:38:58.143 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 5 to smithi17031214-44 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: finishing write tid 6 to smithi17031214-44 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: oid 44 updating version 0 to 341 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: oid 44 updating version 341 to 342 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: oid 44 updating version 342 to 345 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: oid 44 version 345 is already newer than 343 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: oid 44 version 345 is already newer than 344 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 345 (ObjNum 464 snap 110 seq_num 464) dirty exists 2024-03-27T09:38:58.147 INFO:tasks.rados.rados.0.smithi170.stdout:957: left oid 44 (ObjNum 464 snap 110 seq_num 464) 2024-03-27T09:38:58.167 INFO:tasks.rados.rados.0.smithi170.stdout:957: done (2 left) 2024-03-27T09:38:58.167 INFO:tasks.rados.rados.0.smithi170.stdout:960: done (1 left) 2024-03-27T09:38:58.167 INFO:tasks.rados.rados.0.smithi170.stdout:961: write oid 3 current snap is 110 2024-03-27T09:38:58.167 INFO:tasks.rados.rados.0.smithi170.stdout:961: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-03-27T09:38:58.177 INFO:tasks.rados.rados.0.smithi170.stdout:961: writing smithi17031214-3 from 665676 to 1379741 tid 1 2024-03-27T09:38:58.183 INFO:tasks.rados.rados.0.smithi170.stdout:961: writing smithi17031214-3 from 1886514 to 2418000 tid 2 2024-03-27T09:38:58.194 INFO:tasks.rados.rados.0.smithi170.stdout:961: writing smithi17031214-3 from 3169137 to 3865676 tid 3 2024-03-27T09:38:58.194 INFO:tasks.rados.rados.0.smithi170.stdout:962: read oid 9 snap -1 2024-03-27T09:38:58.195 INFO:tasks.rados.rados.0.smithi170.stdout:962: expect (ObjNum 379 snap 93 seq_num 379) 2024-03-27T09:38:58.195 INFO:tasks.rados.rados.0.smithi170.stdout:963: write (excl) oid 10 current snap is 110 2024-03-27T09:38:58.195 INFO:tasks.rados.rados.0.smithi170.stdout:959: finishing write tid 1 to smithi17031214-13 2024-03-27T09:38:58.195 INFO:tasks.rados.rados.0.smithi170.stdout:963: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-03-27T09:38:58.205 INFO:tasks.rados.rados.0.smithi170.stdout:963: writing smithi17031214-10 from 682483 to 1471797 tid 1 2024-03-27T09:38:58.215 INFO:tasks.rados.rados.0.smithi170.stdout:963: writing smithi17031214-10 from 1944996 to 2736493 tid 2 2024-03-27T09:38:58.224 INFO:tasks.rados.rados.0.smithi170.stdout:963: writing smithi17031214-10 from 3196560 to 3751134 tid 3 2024-03-27T09:38:58.224 INFO:tasks.rados.rados.0.smithi170.stdout:964: write oid 21 current snap is 110 2024-03-27T09:38:58.224 INFO:tasks.rados.rados.0.smithi170.stdout:959: finishing write tid 2 to smithi17031214-13 2024-03-27T09:38:58.224 INFO:tasks.rados.rados.0.smithi170.stdout:964: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-03-27T09:38:58.231 INFO:tasks.rados.rados.0.smithi170.stdout:964: writing smithi17031214-21 from 699290 to 1163853 tid 1 2024-03-27T09:38:58.237 INFO:tasks.rados.rados.0.smithi170.stdout:964: writing smithi17031214-21 from 1603478 to 2138633 tid 2 2024-03-27T09:38:58.246 INFO:tasks.rados.rados.0.smithi170.stdout:964: writing smithi17031214-21 from 2823983 to 3589829 tid 3 2024-03-27T09:38:58.246 INFO:tasks.rados.rados.0.smithi170.stdout:965: snap_create 2024-03-27T09:38:58.246 INFO:tasks.rados.rados.0.smithi170.stdout:961: finishing write tid 1 to smithi17031214-3 2024-03-27T09:38:58.246 INFO:tasks.rados.rados.0.smithi170.stdout:959: finishing write tid 3 to smithi17031214-13 2024-03-27T09:38:58.246 INFO:tasks.rados.rados.0.smithi170.stdout:961: finishing write tid 2 to smithi17031214-3 2024-03-27T09:38:58.247 INFO:tasks.rados.rados.0.smithi170.stdout:962: expect (ObjNum 379 snap 93 seq_num 379) 2024-03-27T09:38:58.341 INFO:tasks.rados.rados.0.smithi170.stdout:959: finishing write tid 4 to smithi17031214-13 2024-03-27T09:38:58.341 INFO:tasks.rados.rados.0.smithi170.stdout:959: finishing write tid 5 to smithi17031214-13 2024-03-27T09:38:58.341 INFO:tasks.rados.rados.0.smithi170.stdout:959: oid 13 updating version 0 to 205 2024-03-27T09:38:58.341 INFO:tasks.rados.rados.0.smithi170.stdout:959: oid 13 updating version 205 to 206 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:959: oid 13 version 206 is already newer than 204 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:959: oid 13 updating version 206 to 207 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 207 (ObjNum 466 snap 110 seq_num 466) dirty exists 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:959: left oid 13 (ObjNum 466 snap 110 seq_num 466) 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: finishing write tid 3 to smithi17031214-3 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:963: finishing write tid 1 to smithi17031214-10 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: finishing write tid 4 to smithi17031214-3 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:964: finishing write tid 1 to smithi17031214-21 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: finishing write tid 5 to smithi17031214-3 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: oid 3 updating version 0 to 261 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: oid 3 version 261 is already newer than 260 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: oid 3 updating version 261 to 263 2024-03-27T09:38:58.342 INFO:tasks.rados.rados.0.smithi170.stdout:961: oid 3 version 263 is already newer than 262 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 263 (ObjNum 468 snap 110 seq_num 468) dirty exists 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:961: left oid 3 (ObjNum 468 snap 110 seq_num 468) 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: finishing write tid 2 to smithi17031214-10 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:964: finishing write tid 2 to smithi17031214-21 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:964: finishing write tid 3 to smithi17031214-21 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: finishing write tid 3 to smithi17031214-10 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: finishing write tid 4 to smithi17031214-10 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: finishing write tid 5 to smithi17031214-10 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: oid 10 updating version 0 to 42 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: oid 10 version 42 is already newer than 40 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: oid 10 version 42 is already newer than 39 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: oid 10 version 42 is already newer than 42 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 42 (ObjNum 469 snap 110 seq_num 469) dirty exists 2024-03-27T09:38:58.343 INFO:tasks.rados.rados.0.smithi170.stdout:963: left oid 10 (ObjNum 469 snap 110 seq_num 469) 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: finishing write tid 4 to smithi17031214-21 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: finishing write tid 5 to smithi17031214-21 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: oid 21 updating version 0 to 286 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: oid 21 updating version 286 to 288 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: oid 21 updating version 288 to 289 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: oid 21 version 289 is already newer than 288 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 289 (ObjNum 470 snap 110 seq_num 470) dirty exists 2024-03-27T09:38:58.344 INFO:tasks.rados.rados.0.smithi170.stdout:964: left oid 21 (ObjNum 470 snap 110 seq_num 470) 2024-03-27T09:38:58.976 INFO:tasks.rados.rados.0.smithi170.stdout:959: done (5 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:961: done (4 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:962: done (3 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:963: done (2 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:964: done (1 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:965: done (0 left) 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:966: write oid 46 current snap is 111 2024-03-27T09:38:58.977 INFO:tasks.rados.rados.0.smithi170.stdout:966: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-03-27T09:38:58.983 INFO:tasks.rados.rados.0.smithi170.stdout:966: writing smithi17031214-46 from 716097 to 1255909 tid 1 2024-03-27T09:38:58.992 INFO:tasks.rados.rados.0.smithi170.stdout:966: writing smithi17031214-46 from 1945607 to 2740773 tid 2 2024-03-27T09:38:58.999 INFO:tasks.rados.rados.0.smithi170.stdout:966: writing smithi17031214-46 from 3535053 to 3916097 tid 3 2024-03-27T09:38:59.000 INFO:tasks.rados.rados.0.smithi170.stdout:967: read oid 45 snap 109 2024-03-27T09:38:59.000 INFO:tasks.rados.rados.0.smithi170.stdout:967: expect deleted 2024-03-27T09:38:59.000 INFO:tasks.rados.rados.0.smithi170.stdout:968: snap_create 2024-03-27T09:38:59.070 INFO:tasks.rados.rados.0.smithi170.stdout:966: finishing write tid 1 to smithi17031214-46 2024-03-27T09:38:59.090 INFO:tasks.rados.rados.0.smithi170.stdout:966: finishing write tid 2 to smithi17031214-46 2024-03-27T09:38:59.110 INFO:tasks.rados.rados.0.smithi170.stdout:966: finishing write tid 3 to smithi17031214-46 2024-03-27T09:38:59.113 INFO:tasks.rados.rados.0.smithi170.stdout:966: finishing write tid 4 to smithi17031214-46 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: finishing write tid 5 to smithi17031214-46 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: oid 46 updating version 0 to 382 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: oid 46 version 382 is already newer than 379 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: oid 46 version 382 is already newer than 380 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: oid 46 version 382 is already newer than 382 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 382 (ObjNum 471 snap 111 seq_num 471) dirty exists 2024-03-27T09:38:59.115 INFO:tasks.rados.rados.0.smithi170.stdout:966: left oid 46 (ObjNum 471 snap 111 seq_num 471) 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:966: done (2 left) 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:967: done (1 left) 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:968: done (0 left) 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:969: read oid 26 snap 109 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:969: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:38:59.977 INFO:tasks.rados.rados.0.smithi170.stdout:970: snap_remove snap 111 2024-03-27T09:39:00.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:00.155 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:00.153+0000 7fdad4464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:00.159 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:00.159+0000 7fdad4464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:00.183 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:00.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:00.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:00.304 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:00.307 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:00.978 INFO:tasks.rados.rados.0.smithi170.stdout:970: done (1 left) 2024-03-27T09:39:00.978 INFO:tasks.rados.rados.0.smithi170.stdout:971: delete oid 28 current snap is 112 2024-03-27T09:39:00.978 INFO:tasks.rados.rados.0.smithi170.stdout:969: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:39:01.206 INFO:tasks.rados.rados.0.smithi170.stdout:969: done (1 left) 2024-03-27T09:39:01.206 INFO:tasks.rados.rados.0.smithi170.stdout:971: done (0 left) 2024-03-27T09:39:01.206 INFO:tasks.rados.rados.0.smithi170.stdout:972: snap_remove snap 110 2024-03-27T09:39:01.984 INFO:tasks.rados.rados.0.smithi170.stdout:972: done (0 left) 2024-03-27T09:39:01.984 INFO:tasks.rados.rados.0.smithi170.stdout:973: delete oid 19 current snap is 112 2024-03-27T09:39:02.051 INFO:tasks.rados.rados.0.smithi170.stdout:973: done (0 left) 2024-03-27T09:39:02.051 INFO:tasks.rados.rados.0.smithi170.stdout:974: snap_remove snap 109 2024-03-27T09:39:02.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:02.344 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:02.341+0000 7f07a406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:02.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:02.346+0000 7f07a406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:02.372 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:02.419 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:02.985 INFO:tasks.rados.rados.0.smithi170.stdout:974: done (0 left) 2024-03-27T09:39:02.985 INFO:tasks.rados.rados.0.smithi170.stdout:975: snap_remove snap 108 2024-03-27T09:39:03.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2024-03-27T09:39:03.991 INFO:tasks.rados.rados.0.smithi170.stdout:975: done (0 left) 2024-03-27T09:39:03.991 INFO:tasks.rados.rados.0.smithi170.stdout:976: delete oid 46 current snap is 112 2024-03-27T09:39:04.127 INFO:tasks.rados.rados.0.smithi170.stdout:976: done (0 left) 2024-03-27T09:39:04.127 INFO:tasks.rados.rados.0.smithi170.stdout:977: write (excl) oid 37 current snap is 112 2024-03-27T09:39:04.127 INFO:tasks.rados.rados.0.smithi170.stdout:977: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-03-27T09:39:04.134 INFO:tasks.rados.rados.0.smithi170.stdout:977: writing smithi17031214-37 from 783325 to 1340486 tid 1 2024-03-27T09:39:04.140 INFO:tasks.rados.rados.0.smithi170.stdout:977: writing smithi17031214-37 from 1779535 to 2465686 tid 2 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:977: writing smithi17031214-37 from 3244745 to 3699657 tid 3 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:978: read oid 29 snap -1 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:978: expect (ObjNum 380 snap 93 seq_num 380) 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:979: read oid 10 snap -1 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:979: expect (ObjNum 469 snap 110 seq_num 469) 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:980: read oid 14 snap -1 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:980: expect deleted 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:981: write (excl) oid 34 current snap is 112 2024-03-27T09:39:04.149 INFO:tasks.rados.rados.0.smithi170.stdout:981: seq_num 476 ranges {400132=632410,1438017=546162} 2024-03-27T09:39:04.157 INFO:tasks.rados.rados.0.smithi170.stdout:981: writing smithi17031214-34 from 400132 to 1032542 tid 1 2024-03-27T09:39:04.165 INFO:tasks.rados.rados.0.smithi170.stdout:981: writing smithi17031214-34 from 1438017 to 1984179 tid 2 2024-03-27T09:39:04.165 INFO:tasks.rados.rados.0.smithi170.stdout:982: read oid 47 snap -1 2024-03-27T09:39:04.165 INFO:tasks.rados.rados.0.smithi170.stdout:982: expect (ObjNum 461 snap 108 seq_num 461) 2024-03-27T09:39:04.165 INFO:tasks.rados.rados.0.smithi170.stdout:983: write (excl) oid 20 current snap is 112 2024-03-27T09:39:04.165 INFO:tasks.rados.rados.0.smithi170.stdout:983: seq_num 477 ranges {416939=707659,1896499=120440} 2024-03-27T09:39:04.174 INFO:tasks.rados.rados.0.smithi170.stdout:983: writing smithi17031214-20 from 416939 to 1124598 tid 1 2024-03-27T09:39:04.179 INFO:tasks.rados.rados.0.smithi170.stdout:983: writing smithi17031214-20 from 1896499 to 2016939 tid 2 2024-03-27T09:39:04.180 INFO:tasks.rados.rados.0.smithi170.stdout:984: read oid 8 snap -1 2024-03-27T09:39:04.180 INFO:tasks.rados.rados.0.smithi170.stdout:984: expect (ObjNum 455 snap 105 seq_num 455) 2024-03-27T09:39:04.180 INFO:tasks.rados.rados.0.smithi170.stdout:985: write oid 45 current snap is 112 2024-03-27T09:39:04.180 INFO:tasks.rados.rados.0.smithi170.stdout:985: seq_num 478 ranges {433746=782908,1954981=78765} 2024-03-27T09:39:04.189 INFO:tasks.rados.rados.0.smithi170.stdout:985: writing smithi17031214-45 from 433746 to 1216654 tid 1 2024-03-27T09:39:04.194 INFO:tasks.rados.rados.0.smithi170.stdout:985: writing smithi17031214-45 from 1954981 to 2033746 tid 2 2024-03-27T09:39:04.195 INFO:tasks.rados.rados.0.smithi170.stdout:986: delete oid 21 current snap is 112 2024-03-27T09:39:04.195 INFO:tasks.rados.rados.0.smithi170.stdout:977: finishing write tid 1 to smithi17031214-37 2024-03-27T09:39:04.195 INFO:tasks.rados.rados.0.smithi170.stdout:979: expect (ObjNum 469 snap 110 seq_num 469) 2024-03-27T09:39:04.420 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:04.432 INFO:tasks.rados.rados.0.smithi170.stdout:977: finishing write tid 2 to smithi17031214-37 2024-03-27T09:39:04.432 INFO:tasks.rados.rados.0.smithi170.stdout:978: expect (ObjNum 380 snap 93 seq_num 380) 2024-03-27T09:39:04.530 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:04.528+0000 7f2883714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:04.534 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:04.533+0000 7f2883714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:04.558 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:981: finishing write tid 1 to smithi17031214-34 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:977: finishing write tid 3 to smithi17031214-37 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:978: done (9 left) 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:979: done (8 left) 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:980: done (7 left) 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:986: done (6 left) 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:987: write oid 12 current snap is 112 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:983: finishing write tid 1 to smithi17031214-20 2024-03-27T09:39:04.590 INFO:tasks.rados.rados.0.smithi170.stdout:987: seq_num 480 ranges {467360=649759,1671945=395415} 2024-03-27T09:39:04.595 INFO:tasks.rados.rados.0.smithi170.stdout:987: writing smithi17031214-12 from 467360 to 1117119 tid 1 2024-03-27T09:39:04.601 INFO:tasks.rados.rados.0.smithi170.stdout:987: writing smithi17031214-12 from 1671945 to 2067360 tid 2 2024-03-27T09:39:04.601 INFO:tasks.rados.rados.0.smithi170.stdout:988: delete oid 48 current snap is 112 2024-03-27T09:39:04.601 INFO:tasks.rados.rados.0.smithi170.stdout:977: finishing write tid 4 to smithi17031214-37 2024-03-27T09:39:04.601 INFO:tasks.rados.rados.0.smithi170.stdout:977: finishing write tid 5 to smithi17031214-37 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:977: oid 37 updating version 0 to 336 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:977: oid 37 updating version 336 to 338 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:977: oid 37 version 338 is already newer than 335 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:977: oid 37 version 338 is already newer than 338 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 338 (ObjNum 475 snap 112 seq_num 475) dirty exists 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:977: left oid 37 (ObjNum 475 snap 112 seq_num 475) 2024-03-27T09:39:04.602 INFO:tasks.rados.rados.0.smithi170.stdout:984: expect (ObjNum 455 snap 105 seq_num 455) 2024-03-27T09:39:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:04.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:04.829 INFO:tasks.rados.rados.0.smithi170.stdout:982: expect (ObjNum 461 snap 108 seq_num 461) 2024-03-27T09:39:04.968 INFO:tasks.rados.rados.0.smithi170.stdout:983: finishing write tid 2 to smithi17031214-20 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:985: finishing write tid 1 to smithi17031214-45 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: finishing write tid 2 to smithi17031214-34 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: finishing write tid 3 to smithi17031214-20 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: finishing write tid 4 to smithi17031214-20 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: oid 20 updating version 0 to 311 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: oid 20 updating version 311 to 312 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: oid 20 version 312 is already newer than 310 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 312 (ObjNum 477 snap 112 seq_num 477) dirty exists 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:983: left oid 20 (ObjNum 477 snap 112 seq_num 477) 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: finishing write tid 3 to smithi17031214-34 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: finishing write tid 4 to smithi17031214-34 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: oid 34 updating version 0 to 388 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: oid 34 updating version 388 to 390 2024-03-27T09:39:04.969 INFO:tasks.rados.rados.0.smithi170.stdout:981: oid 34 version 390 is already newer than 389 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 390 (ObjNum 476 snap 112 seq_num 476) dirty exists 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:981: left oid 34 (ObjNum 476 snap 112 seq_num 476) 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: finishing write tid 2 to smithi17031214-45 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: finishing write tid 3 to smithi17031214-45 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: finishing write tid 4 to smithi17031214-45 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: oid 45 updating version 0 to 331 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: oid 45 updating version 331 to 332 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: oid 45 version 332 is already newer than 330 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 332 (ObjNum 478 snap 112 seq_num 478) dirty exists 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:985: left oid 45 (ObjNum 478 snap 112 seq_num 478) 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:977: done (7 left) 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:981: done (6 left) 2024-03-27T09:39:04.970 INFO:tasks.rados.rados.0.smithi170.stdout:982: done (5 left) 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:983: done (4 left) 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:984: done (3 left) 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:985: done (2 left) 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:988: done (1 left) 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:989: snap_create 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: finishing write tid 1 to smithi17031214-12 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: finishing write tid 2 to smithi17031214-12 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: finishing write tid 3 to smithi17031214-12 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: finishing write tid 4 to smithi17031214-12 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: oid 12 updating version 0 to 314 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: oid 12 updating version 314 to 315 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: oid 12 version 315 is already newer than 313 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 315 (ObjNum 480 snap 112 seq_num 480) dirty exists 2024-03-27T09:39:04.971 INFO:tasks.rados.rados.0.smithi170.stdout:987: left oid 12 (ObjNum 480 snap 112 seq_num 480) 2024-03-27T09:39:04.994 INFO:tasks.rados.rados.0.smithi170.stdout:987: done (1 left) 2024-03-27T09:39:04.994 INFO:tasks.rados.rados.0.smithi170.stdout:989: done (0 left) 2024-03-27T09:39:04.995 INFO:tasks.rados.rados.0.smithi170.stdout:990: write (excl) oid 41 current snap is 113 2024-03-27T09:39:04.995 INFO:tasks.rados.rados.0.smithi170.stdout:990: seq_num 482 ranges {500974=400257,1388909=557169} 2024-03-27T09:39:04.999 INFO:tasks.rados.rados.0.smithi170.stdout:990: writing smithi17031214-41 from 500974 to 901231 tid 1 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:990: writing smithi17031214-41 from 1388909 to 1946078 tid 2 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:991: read oid 17 snap -1 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:991: expect deleted 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:992: read oid 39 snap -1 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:992: expect (ObjNum 403 snap 98 seq_num 403) 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:993: read oid 18 snap -1 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:993: expect deleted 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:994: read oid 6 snap 112 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:994: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:995: read oid 30 snap 112 2024-03-27T09:39:05.009 INFO:tasks.rados.rados.0.smithi170.stdout:995: expect (ObjNum 445 snap 103 seq_num 445) 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:996: read oid 1 snap -1 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:996: expect deleted 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:997: read oid 23 snap -1 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:997: expect deleted 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:998: write (excl) oid 34 current snap is 113 2024-03-27T09:39:05.010 INFO:tasks.rados.rados.0.smithi170.stdout:998: seq_num 483 ranges {517781=475506,1731038=386743} 2024-03-27T09:39:05.015 INFO:tasks.rados.rados.0.smithi170.stdout:998: writing smithi17031214-34 from 517781 to 993287 tid 1 2024-03-27T09:39:05.024 INFO:tasks.rados.rados.0.smithi170.stdout:998: writing smithi17031214-34 from 1731038 to 2117781 tid 2 2024-03-27T09:39:05.024 INFO:tasks.rados.rados.0.smithi170.stdout:999: write oid 45 current snap is 113 2024-03-27T09:39:05.024 INFO:tasks.rados.rados.0.smithi170.stdout:999: seq_num 484 ranges {534588=550755,1789520=345068} 2024-03-27T09:39:05.033 INFO:tasks.rados.rados.0.smithi170.stdout:999: writing smithi17031214-45 from 534588 to 1085343 tid 1 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:999: writing smithi17031214-45 from 1789520 to 2134588 tid 2 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:1000: read oid 4 snap -1 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:1000: expect (ObjNum 419 snap 99 seq_num 419) 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:1001: read oid 44 snap -1 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:1001: expect (ObjNum 464 snap 110 seq_num 464) 2024-03-27T09:39:05.038 INFO:tasks.rados.rados.0.smithi170.stdout:1002: delete oid 36 current snap is 113 2024-03-27T09:39:05.114 INFO:tasks.rados.rados.0.smithi170.stdout:992: expect (ObjNum 403 snap 98 seq_num 403) 2024-03-27T09:39:05.248 INFO:tasks.rados.rados.0.smithi170.stdout:1000: expect (ObjNum 419 snap 99 seq_num 419) 2024-03-27T09:39:05.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:05.405 INFO:tasks.rados.rados.0.smithi170.stdout:990: finishing write tid 1 to smithi17031214-41 2024-03-27T09:39:05.405 INFO:tasks.rados.rados.0.smithi170.stdout:991: done (12 left) 2024-03-27T09:39:05.405 INFO:tasks.rados.rados.0.smithi170.stdout:992: done (11 left) 2024-03-27T09:39:05.406 INFO:tasks.rados.rados.0.smithi170.stdout:997: done (10 left) 2024-03-27T09:39:05.406 INFO:tasks.rados.rados.0.smithi170.stdout:1000: done (9 left) 2024-03-27T09:39:05.406 INFO:tasks.rados.rados.0.smithi170.stdout:1002: done (8 left) 2024-03-27T09:39:05.406 INFO:tasks.rados.rados.0.smithi170.stdout:1003: write (excl) oid 43 current snap is 113 2024-03-27T09:39:05.406 INFO:tasks.rados.rados.0.smithi170.stdout:995: expect (ObjNum 445 snap 103 seq_num 445) 2024-03-27T09:39:05.407 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:05.410 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:05.613 INFO:tasks.rados.rados.0.smithi170.stdout:994: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:05.736 INFO:tasks.rados.rados.0.smithi170.stdout:999: finishing write tid 1 to smithi17031214-45 2024-03-27T09:39:05.736 INFO:tasks.rados.rados.0.smithi170.stdout:1001: expect (ObjNum 464 snap 110 seq_num 464) 2024-03-27T09:39:05.931 INFO:tasks.rados.rados.0.smithi170.stdout:998: finishing write tid 1 to smithi17031214-34 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: finishing write tid 2 to smithi17031214-41 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:999: finishing write tid 2 to smithi17031214-45 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: finishing write tid 3 to smithi17031214-41 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: finishing write tid 4 to smithi17031214-41 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: oid 41 updating version 0 to 340 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: oid 41 updating version 340 to 342 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: oid 41 version 342 is already newer than 341 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 342 (ObjNum 482 snap 113 seq_num 482) dirty exists 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:990: left oid 41 (ObjNum 482 snap 113 seq_num 482) 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:998: finishing write tid 2 to smithi17031214-34 2024-03-27T09:39:05.932 INFO:tasks.rados.rados.0.smithi170.stdout:1003: seq_num 486 ranges {568202=701253,1906484=261718} 2024-03-27T09:39:05.938 INFO:tasks.rados.rados.0.smithi170.stdout:1003: writing smithi17031214-43 from 568202 to 1269455 tid 1 2024-03-27T09:39:05.943 INFO:tasks.rados.rados.0.smithi170.stdout:1003: writing smithi17031214-43 from 1906484 to 2168202 tid 2 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:990: done (8 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:993: done (7 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:994: done (6 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:995: done (5 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:996: done (4 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:1001: done (3 left) 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:1004: read oid 48 snap 112 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:1004: expect deleted 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:1005: delete oid 29 current snap is 113 2024-03-27T09:39:05.944 INFO:tasks.rados.rados.0.smithi170.stdout:999: finishing write tid 3 to smithi17031214-45 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:999: finishing write tid 4 to smithi17031214-45 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:999: oid 45 updating version 0 to 336 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:999: oid 45 version 336 is already newer than 335 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:999: oid 45 version 336 is already newer than 334 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 336 (ObjNum 484 snap 113 seq_num 484) dirty exists 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:999: left oid 45 (ObjNum 484 snap 113 seq_num 484) 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: finishing write tid 3 to smithi17031214-34 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: finishing write tid 4 to smithi17031214-34 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: oid 34 updating version 0 to 393 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: oid 34 version 393 is already newer than 392 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: oid 34 updating version 393 to 394 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 394 (ObjNum 483 snap 113 seq_num 483) dirty exists 2024-03-27T09:39:05.945 INFO:tasks.rados.rados.0.smithi170.stdout:998: left oid 34 (ObjNum 483 snap 113 seq_num 483) 2024-03-27T09:39:05.971 INFO:tasks.rados.rados.0.smithi170.stdout:1003: finishing write tid 1 to smithi17031214-43 2024-03-27T09:39:06.001 INFO:tasks.rados.rados.0.smithi170.stdout:1003: finishing write tid 2 to smithi17031214-43 2024-03-27T09:39:06.007 INFO:tasks.rados.rados.0.smithi170.stdout:998: done (4 left) 2024-03-27T09:39:06.008 INFO:tasks.rados.rados.0.smithi170.stdout:999: done (3 left) 2024-03-27T09:39:06.008 INFO:tasks.rados.rados.0.smithi170.stdout:1004: done (2 left) 2024-03-27T09:39:06.008 INFO:tasks.rados.rados.0.smithi170.stdout:1005: done (1 left) 2024-03-27T09:39:06.008 INFO:tasks.rados.rados.0.smithi170.stdout:1006: snap_remove snap 112 2024-03-27T09:39:06.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:06.718 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:06.716+0000 7f12bb79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:06.722 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:06.722+0000 7f12bb79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:06.746 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:06.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:07.001 INFO:tasks.rados.rados.0.smithi170.stdout:1006: done (1 left) 2024-03-27T09:39:07.001 INFO:tasks.rados.rados.0.smithi170.stdout:1007: write oid 3 current snap is 113 2024-03-27T09:39:07.001 INFO:tasks.rados.rados.0.smithi170.stdout:1003: finishing write tid 3 to smithi17031214-43 2024-03-27T09:39:07.001 INFO:tasks.rados.rados.0.smithi170.stdout:1007: seq_num 488 ranges {601816=568104,1623448=451823} 2024-03-27T09:39:07.008 INFO:tasks.rados.rados.0.smithi170.stdout:1007: writing smithi17031214-3 from 601816 to 1169920 tid 1 2024-03-27T09:39:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:1007: writing smithi17031214-3 from 1623448 to 2075271 tid 2 2024-03-27T09:39:07.013 INFO:tasks.rados.rados.0.smithi170.stdout:1008: write (excl) oid 11 current snap is 113 2024-03-27T09:39:07.014 INFO:tasks.rados.rados.0.smithi170.stdout:1008: seq_num 489 ranges {618623=643353,1681930=536693} 2024-03-27T09:39:07.023 INFO:tasks.rados.rados.0.smithi170.stdout:1008: writing smithi17031214-11 from 618623 to 1261976 tid 1 2024-03-27T09:39:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:1008: writing smithi17031214-11 from 1681930 to 2218623 tid 2 2024-03-27T09:39:07.031 INFO:tasks.rados.rados.0.smithi170.stdout:1009: read oid 50 snap -1 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1009: expect deleted 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1010: write oid 27 current snap is 113 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1003: finishing write tid 4 to smithi17031214-43 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1003: oid 43 updating version 0 to 337 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1003: oid 43 updating version 337 to 338 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1003: oid 43 updating version 338 to 339 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 339 (ObjNum 486 snap 113 seq_num 486) dirty exists 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1003: left oid 43 (ObjNum 486 snap 113 seq_num 486) 2024-03-27T09:39:07.032 INFO:tasks.rados.rados.0.smithi170.stdout:1010: seq_num 490 ranges {635430=718602,2140412=95018} 2024-03-27T09:39:07.042 INFO:tasks.rados.rados.0.smithi170.stdout:1010: writing smithi17031214-27 from 635430 to 1354032 tid 1 2024-03-27T09:39:07.047 INFO:tasks.rados.rados.0.smithi170.stdout:1010: writing smithi17031214-27 from 2140412 to 2235430 tid 2 2024-03-27T09:39:07.047 INFO:tasks.rados.rados.0.smithi170.stdout:1003: done (4 left) 2024-03-27T09:39:07.047 INFO:tasks.rados.rados.0.smithi170.stdout:1011: write oid 5 current snap is 113 2024-03-27T09:39:07.048 INFO:tasks.rados.rados.0.smithi170.stdout:1011: seq_num 491 ranges {652237=793851,2198894=53343} 2024-03-27T09:39:07.059 INFO:tasks.rados.rados.0.smithi170.stdout:1011: writing smithi17031214-5 from 652237 to 1446088 tid 1 2024-03-27T09:39:07.064 INFO:tasks.rados.rados.0.smithi170.stdout:1011: writing smithi17031214-5 from 2198894 to 2252237 tid 2 2024-03-27T09:39:07.064 INFO:tasks.rados.rados.0.smithi170.stdout:1012: delete oid 7 current snap is 113 2024-03-27T09:39:07.096 INFO:tasks.rados.rados.0.smithi170.stdout:1007: finishing write tid 1 to smithi17031214-3 2024-03-27T09:39:07.120 INFO:tasks.rados.rados.0.smithi170.stdout:1007: finishing write tid 2 to smithi17031214-3 2024-03-27T09:39:07.194 INFO:tasks.rados.rados.0.smithi170.stdout:1008: finishing write tid 1 to smithi17031214-11 2024-03-27T09:39:07.198 INFO:tasks.rados.rados.0.smithi170.stdout:1007: finishing write tid 3 to smithi17031214-3 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:1007: finishing write tid 4 to smithi17031214-3 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:1007: oid 3 updating version 0 to 268 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:1007: oid 3 version 268 is already newer than 268 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:1007: oid 3 updating version 268 to 269 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 269 (ObjNum 488 snap 113 seq_num 488) dirty exists 2024-03-27T09:39:07.201 INFO:tasks.rados.rados.0.smithi170.stdout:1007: left oid 3 (ObjNum 488 snap 113 seq_num 488) 2024-03-27T09:39:07.230 INFO:tasks.rados.rados.0.smithi170.stdout:1010: finishing write tid 1 to smithi17031214-27 2024-03-27T09:39:07.232 INFO:tasks.rados.rados.0.smithi170.stdout:1007: done (5 left) 2024-03-27T09:39:07.232 INFO:tasks.rados.rados.0.smithi170.stdout:1009: done (4 left) 2024-03-27T09:39:07.232 INFO:tasks.rados.rados.0.smithi170.stdout:1012: done (3 left) 2024-03-27T09:39:07.232 INFO:tasks.rados.rados.0.smithi170.stdout:1013: delete oid 45 current snap is 113 2024-03-27T09:39:07.232 INFO:tasks.rados.rados.0.smithi170.stdout:1011: finishing write tid 1 to smithi17031214-5 2024-03-27T09:39:07.248 INFO:tasks.rados.rados.0.smithi170.stdout:1008: finishing write tid 2 to smithi17031214-11 2024-03-27T09:39:07.249 INFO:tasks.rados.rados.0.smithi170.stdout:1008: finishing write tid 3 to smithi17031214-11 2024-03-27T09:39:07.253 INFO:tasks.rados.rados.0.smithi170.stdout:1008: finishing write tid 4 to smithi17031214-11 2024-03-27T09:39:07.254 INFO:tasks.rados.rados.0.smithi170.stdout:1008: oid 11 updating version 0 to 320 2024-03-27T09:39:07.254 INFO:tasks.rados.rados.0.smithi170.stdout:1008: oid 11 updating version 320 to 322 2024-03-27T09:39:07.254 INFO:tasks.rados.rados.0.smithi170.stdout:1008: oid 11 version 322 is already newer than 322 2024-03-27T09:39:07.254 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 322 (ObjNum 489 snap 113 seq_num 489) dirty exists 2024-03-27T09:39:07.254 INFO:tasks.rados.rados.0.smithi170.stdout:1008: left oid 11 (ObjNum 489 snap 113 seq_num 489) 2024-03-27T09:39:07.265 INFO:tasks.rados.rados.0.smithi170.stdout:1010: finishing write tid 2 to smithi17031214-27 2024-03-27T09:39:07.270 INFO:tasks.rados.rados.0.smithi170.stdout:1011: finishing write tid 2 to smithi17031214-5 2024-03-27T09:39:07.289 INFO:tasks.rados.rados.0.smithi170.stdout:1011: finishing write tid 3 to smithi17031214-5 2024-03-27T09:39:07.289 INFO:tasks.rados.rados.0.smithi170.stdout:1010: finishing write tid 3 to smithi17031214-27 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:1011: finishing write tid 4 to smithi17031214-5 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:1011: oid 5 updating version 0 to 167 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:1011: oid 5 version 167 is already newer than 165 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:1011: oid 5 version 167 is already newer than 166 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 167 (ObjNum 491 snap 113 seq_num 491) dirty exists 2024-03-27T09:39:07.290 INFO:tasks.rados.rados.0.smithi170.stdout:1011: left oid 5 (ObjNum 491 snap 113 seq_num 491) 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:1010: finishing write tid 4 to smithi17031214-27 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:1010: oid 27 updating version 0 to 344 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:1010: oid 27 version 344 is already newer than 343 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:1010: oid 27 version 344 is already newer than 342 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 344 (ObjNum 490 snap 113 seq_num 490) dirty exists 2024-03-27T09:39:07.294 INFO:tasks.rados.rados.0.smithi170.stdout:1010: left oid 27 (ObjNum 490 snap 113 seq_num 490) 2024-03-27T09:39:07.309 INFO:tasks.rados.rados.0.smithi170.stdout:1008: done (3 left) 2024-03-27T09:39:07.309 INFO:tasks.rados.rados.0.smithi170.stdout:1010: done (2 left) 2024-03-27T09:39:07.309 INFO:tasks.rados.rados.0.smithi170.stdout:1011: done (1 left) 2024-03-27T09:39:07.309 INFO:tasks.rados.rados.0.smithi170.stdout:1013: done (0 left) 2024-03-27T09:39:07.309 INFO:tasks.rados.rados.0.smithi170.stdout:1014: delete oid 46 current snap is 113 2024-03-27T09:39:07.316 INFO:tasks.rados.rados.0.smithi170.stdout:1014: done (0 left) 2024-03-27T09:39:07.316 INFO:tasks.rados.rados.0.smithi170.stdout:1015: delete oid 47 current snap is 113 2024-03-27T09:39:07.336 INFO:tasks.rados.rados.0.smithi170.stdout:1015: done (0 left) 2024-03-27T09:39:07.336 INFO:tasks.rados.rados.0.smithi170.stdout:1016: delete oid 44 current snap is 113 2024-03-27T09:39:07.357 INFO:tasks.rados.rados.0.smithi170.stdout:1016: done (0 left) 2024-03-27T09:39:07.357 INFO:tasks.rados.rados.0.smithi170.stdout:1017: write (excl) oid 14 current snap is 113 2024-03-27T09:39:07.357 INFO:tasks.rados.rados.0.smithi170.stdout:1017: seq_num 497 ranges {753079=561698,2033433=319646} 2024-03-27T09:39:07.364 INFO:tasks.rados.rados.0.smithi170.stdout:1017: writing smithi17031214-14 from 753079 to 1314777 tid 1 2024-03-27T09:39:07.370 INFO:tasks.rados.rados.0.smithi170.stdout:1017: writing smithi17031214-14 from 2033433 to 2353079 tid 2 2024-03-27T09:39:07.370 INFO:tasks.rados.rados.0.smithi170.stdout:1018: snap_create 2024-03-27T09:39:07.385 INFO:tasks.rados.rados.0.smithi170.stdout:1017: finishing write tid 1 to smithi17031214-14 2024-03-27T09:39:07.401 INFO:tasks.rados.rados.0.smithi170.stdout:1017: finishing write tid 2 to smithi17031214-14 2024-03-27T09:39:07.409 INFO:tasks.rados.rados.0.smithi170.stdout:1017: finishing write tid 3 to smithi17031214-14 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:1017: finishing write tid 4 to smithi17031214-14 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:1017: oid 14 updating version 0 to 209 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:1017: oid 14 version 209 is already newer than 208 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:1017: oid 14 updating version 209 to 210 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 210 (ObjNum 497 snap 113 seq_num 497) dirty exists 2024-03-27T09:39:07.411 INFO:tasks.rados.rados.0.smithi170.stdout:1017: left oid 14 (ObjNum 497 snap 113 seq_num 497) 2024-03-27T09:39:08.004 INFO:tasks.rados.rados.0.smithi170.stdout:1017: done (1 left) 2024-03-27T09:39:08.004 INFO:tasks.rados.rados.0.smithi170.stdout:1018: done (0 left) 2024-03-27T09:39:08.004 INFO:tasks.rados.rados.0.smithi170.stdout:1019: delete oid 1 current snap is 114 2024-03-27T09:39:08.049 INFO:tasks.rados.rados.0.smithi170.stdout:1019: done (0 left) 2024-03-27T09:39:08.049 INFO:tasks.rados.rados.0.smithi170.stdout:1020: write oid 25 current snap is 114 2024-03-27T09:39:08.049 INFO:tasks.rados.rados.0.smithi170.stdout:1020: seq_num 499 ranges {786693=712196,2150397=236296} 2024-03-27T09:39:08.057 INFO:tasks.rados.rados.0.smithi170.stdout:1020: writing smithi17031214-25 from 786693 to 1498889 tid 1 2024-03-27T09:39:08.063 INFO:tasks.rados.rados.0.smithi170.stdout:1020: writing smithi17031214-25 from 2150397 to 2386693 tid 2 2024-03-27T09:39:08.063 INFO:tasks.rados.rados.0.smithi170.stdout:1021: write oid 37 current snap is 114 2024-03-27T09:39:08.063 INFO:tasks.rados.rados.0.smithi170.stdout:1021: seq_num 500 ranges {403500=787445,1692526=473837} 2024-03-27T09:39:08.068 INFO:tasks.rados.rados.0.smithi170.stdout:1021: writing smithi17031214-37 from 403500 to 1190945 tid 1 2024-03-27T09:39:08.074 INFO:tasks.rados.rados.0.smithi170.stdout:1021: writing smithi17031214-37 from 1692526 to 2166363 tid 2 2024-03-27T09:39:08.074 INFO:tasks.rados.rados.0.smithi170.stdout:1022: snap_create 2024-03-27T09:39:08.095 INFO:tasks.rados.rados.0.smithi170.stdout:1020: finishing write tid 1 to smithi17031214-25 2024-03-27T09:39:08.111 INFO:tasks.rados.rados.0.smithi170.stdout:1020: finishing write tid 2 to smithi17031214-25 2024-03-27T09:39:08.112 INFO:tasks.rados.rados.0.smithi170.stdout:1020: finishing write tid 3 to smithi17031214-25 2024-03-27T09:39:08.115 INFO:tasks.rados.rados.0.smithi170.stdout:1020: finishing write tid 4 to smithi17031214-25 2024-03-27T09:39:08.115 INFO:tasks.rados.rados.0.smithi170.stdout:1020: oid 25 updating version 0 to 349 2024-03-27T09:39:08.115 INFO:tasks.rados.rados.0.smithi170.stdout:1020: oid 25 version 349 is already newer than 349 2024-03-27T09:39:08.115 INFO:tasks.rados.rados.0.smithi170.stdout:1020: oid 25 version 349 is already newer than 347 2024-03-27T09:39:08.115 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 349 (ObjNum 499 snap 114 seq_num 499) dirty exists 2024-03-27T09:39:08.116 INFO:tasks.rados.rados.0.smithi170.stdout:1020: left oid 25 (ObjNum 499 snap 114 seq_num 499) 2024-03-27T09:39:08.136 INFO:tasks.rados.rados.0.smithi170.stdout:1021: finishing write tid 1 to smithi17031214-37 2024-03-27T09:39:08.154 INFO:tasks.rados.rados.0.smithi170.stdout:1021: finishing write tid 2 to smithi17031214-37 2024-03-27T09:39:08.157 INFO:tasks.rados.rados.0.smithi170.stdout:1021: finishing write tid 3 to smithi17031214-37 2024-03-27T09:39:08.160 INFO:tasks.rados.rados.0.smithi170.stdout:1021: finishing write tid 4 to smithi17031214-37 2024-03-27T09:39:08.160 INFO:tasks.rados.rados.0.smithi170.stdout:1021: oid 37 updating version 0 to 346 2024-03-27T09:39:08.160 INFO:tasks.rados.rados.0.smithi170.stdout:1021: oid 37 updating version 346 to 348 2024-03-27T09:39:08.160 INFO:tasks.rados.rados.0.smithi170.stdout:1021: oid 37 version 348 is already newer than 348 2024-03-27T09:39:08.160 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 348 (ObjNum 500 snap 114 seq_num 500) dirty exists 2024-03-27T09:39:08.161 INFO:tasks.rados.rados.0.smithi170.stdout:1021: left oid 37 (ObjNum 500 snap 114 seq_num 500) 2024-03-27T09:39:08.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2024-03-27T09:39:08.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:08.906 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:08.903+0000 7fb28759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:08.909 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:08.909+0000 7fb28759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:08.933 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:08.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:09.007 INFO:tasks.rados.rados.0.smithi170.stdout:1020: done (2 left) 2024-03-27T09:39:09.007 INFO:tasks.rados.rados.0.smithi170.stdout:1021: done (1 left) 2024-03-27T09:39:09.007 INFO:tasks.rados.rados.0.smithi170.stdout:1022: done (0 left) 2024-03-27T09:39:09.007 INFO:tasks.rados.rados.0.smithi170.stdout:1023: write oid 23 current snap is 115 2024-03-27T09:39:09.007 INFO:tasks.rados.rados.0.smithi170.stdout:1023: seq_num 501 ranges {420307=462694,1351008=617495} 2024-03-27T09:39:09.011 INFO:tasks.rados.rados.0.smithi170.stdout:1023: writing smithi17031214-23 from 420307 to 883001 tid 1 2024-03-27T09:39:09.017 INFO:tasks.rados.rados.0.smithi170.stdout:1023: writing smithi17031214-23 from 1351008 to 1968503 tid 2 2024-03-27T09:39:09.018 INFO:tasks.rados.rados.0.smithi170.stdout:1024: snap_create 2024-03-27T09:39:09.066 INFO:tasks.rados.rados.0.smithi170.stdout:1023: finishing write tid 1 to smithi17031214-23 2024-03-27T09:39:09.087 INFO:tasks.rados.rados.0.smithi170.stdout:1023: finishing write tid 2 to smithi17031214-23 2024-03-27T09:39:09.096 INFO:tasks.rados.rados.0.smithi170.stdout:1023: finishing write tid 3 to smithi17031214-23 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:1023: finishing write tid 4 to smithi17031214-23 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:1023: oid 23 updating version 0 to 399 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:1023: oid 23 updating version 399 to 400 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:1023: oid 23 version 400 is already newer than 398 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 400 (ObjNum 501 snap 115 seq_num 501) dirty exists 2024-03-27T09:39:09.098 INFO:tasks.rados.rados.0.smithi170.stdout:1023: left oid 23 (ObjNum 501 snap 115 seq_num 501) 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1023: done (1 left) 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1024: done (0 left) 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1025: read oid 17 snap -1 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1025: expect deleted 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1026: read oid 50 snap -1 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1026: expect deleted 2024-03-27T09:39:10.012 INFO:tasks.rados.rados.0.smithi170.stdout:1027: write oid 48 current snap is 116 2024-03-27T09:39:10.013 INFO:tasks.rados.rados.0.smithi170.stdout:1027: seq_num 502 ranges {437114=654296,1525843=477506} 2024-03-27T09:39:10.018 INFO:tasks.rados.rados.0.smithi170.stdout:1027: writing smithi17031214-48 from 437114 to 1091410 tid 1 2024-03-27T09:39:10.024 INFO:tasks.rados.rados.0.smithi170.stdout:1027: writing smithi17031214-48 from 1525843 to 2003349 tid 2 2024-03-27T09:39:10.024 INFO:tasks.rados.rados.0.smithi170.stdout:1028: write oid 30 current snap is 116 2024-03-27T09:39:10.024 INFO:tasks.rados.rados.0.smithi170.stdout:1028: seq_num 503 ranges {453921=729545,1584325=621164} 2024-03-27T09:39:10.030 INFO:tasks.rados.rados.0.smithi170.stdout:1028: writing smithi17031214-30 from 453921 to 1183466 tid 1 2024-03-27T09:39:10.036 INFO:tasks.rados.rados.0.smithi170.stdout:1028: writing smithi17031214-30 from 1584325 to 2205489 tid 2 2024-03-27T09:39:10.036 INFO:tasks.rados.rados.0.smithi170.stdout:1029: snap_remove snap 114 2024-03-27T09:39:10.411 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:10.513 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:10.516 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:10.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1029: done (4 left) 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1030: read oid 2 snap -1 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1030: expect deleted 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1031: read oid 29 snap -1 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1031: expect deleted 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1032: read oid 32 snap 113 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1032: expect (ObjNum 312 snap 81 seq_num 312) 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1033: delete oid 31 current snap is 116 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1027: finishing write tid 1 to smithi17031214-48 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1028: finishing write tid 1 to smithi17031214-30 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1027: finishing write tid 2 to smithi17031214-48 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1028: finishing write tid 2 to smithi17031214-30 2024-03-27T09:39:11.016 INFO:tasks.rados.rados.0.smithi170.stdout:1027: finishing write tid 3 to smithi17031214-48 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1027: finishing write tid 4 to smithi17031214-48 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1027: oid 48 updating version 0 to 270 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1027: oid 48 updating version 270 to 272 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1027: oid 48 version 272 is already newer than 271 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 272 (ObjNum 502 snap 116 seq_num 502) dirty exists 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1027: left oid 48 (ObjNum 502 snap 116 seq_num 502) 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: finishing write tid 3 to smithi17031214-30 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: finishing write tid 4 to smithi17031214-30 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: oid 30 updating version 0 to 213 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: oid 30 version 213 is already newer than 213 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: oid 30 updating version 213 to 214 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 214 (ObjNum 503 snap 116 seq_num 503) dirty exists 2024-03-27T09:39:11.017 INFO:tasks.rados.rados.0.smithi170.stdout:1028: left oid 30 (ObjNum 503 snap 116 seq_num 503) 2024-03-27T09:39:11.079 INFO:tasks.rados.rados.0.smithi170.stdout:1025: done (7 left) 2024-03-27T09:39:11.079 INFO:tasks.rados.rados.0.smithi170.stdout:1026: done (6 left) 2024-03-27T09:39:11.079 INFO:tasks.rados.rados.0.smithi170.stdout:1027: done (5 left) 2024-03-27T09:39:11.079 INFO:tasks.rados.rados.0.smithi170.stdout:1028: done (4 left) 2024-03-27T09:39:11.080 INFO:tasks.rados.rados.0.smithi170.stdout:1030: done (3 left) 2024-03-27T09:39:11.080 INFO:tasks.rados.rados.0.smithi170.stdout:1031: done (2 left) 2024-03-27T09:39:11.080 INFO:tasks.rados.rados.0.smithi170.stdout:1033: done (1 left) 2024-03-27T09:39:11.080 INFO:tasks.rados.rados.0.smithi170.stdout:1034: write oid 4 current snap is 116 2024-03-27T09:39:11.080 INFO:tasks.rados.rados.0.smithi170.stdout:1034: seq_num 505 ranges {487535=480043,1584936=624833} 2024-03-27T09:39:11.086 INFO:tasks.rados.rados.0.smithi170.stdout:1034: writing smithi17031214-4 from 487535 to 967578 tid 1 2024-03-27T09:39:11.095 INFO:tasks.rados.rados.0.smithi170.stdout:1034: writing smithi17031214-4 from 1584936 to 2209769 tid 2 2024-03-27T09:39:11.095 INFO:tasks.rados.rados.0.smithi170.stdout:1035: read oid 11 snap -1 2024-03-27T09:39:11.095 INFO:tasks.rados.rados.0.smithi170.stdout:1035: expect (ObjNum 489 snap 113 seq_num 489) 2024-03-27T09:39:11.095 INFO:tasks.rados.rados.0.smithi170.stdout:1036: snap_remove snap 115 2024-03-27T09:39:11.118 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:11.115+0000 7ff161a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:11.121 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:11.121+0000 7ff161a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:11.154 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:11.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:12.018 INFO:tasks.rados.rados.0.smithi170.stdout:1036: done (3 left) 2024-03-27T09:39:12.018 INFO:tasks.rados.rados.0.smithi170.stdout:1037: write oid 9 current snap is 116 2024-03-27T09:39:12.018 INFO:tasks.rados.rados.0.smithi170.stdout:1037: seq_num 506 ranges {504342=555292,1643418=484844} 2024-03-27T09:39:12.023 INFO:tasks.rados.rados.0.smithi170.stdout:1037: writing smithi17031214-9 from 504342 to 1059634 tid 1 2024-03-27T09:39:12.030 INFO:tasks.rados.rados.0.smithi170.stdout:1037: writing smithi17031214-9 from 1643418 to 2128262 tid 2 2024-03-27T09:39:12.030 INFO:tasks.rados.rados.0.smithi170.stdout:1038: delete oid 47 current snap is 116 2024-03-27T09:39:12.030 INFO:tasks.rados.rados.0.smithi170.stdout:1032: expect (ObjNum 312 snap 81 seq_num 312) 2024-03-27T09:39:12.257 INFO:tasks.rados.rados.0.smithi170.stdout:1034: finishing write tid 1 to smithi17031214-4 2024-03-27T09:39:12.257 INFO:tasks.rados.rados.0.smithi170.stdout:1032: done (4 left) 2024-03-27T09:39:12.257 INFO:tasks.rados.rados.0.smithi170.stdout:1038: done (3 left) 2024-03-27T09:39:12.257 INFO:tasks.rados.rados.0.smithi170.stdout:1039: write oid 21 current snap is 116 2024-03-27T09:39:12.257 INFO:tasks.rados.rados.0.smithi170.stdout:1035: expect (ObjNum 489 snap 113 seq_num 489) 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: finishing write tid 2 to smithi17031214-4 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: finishing write tid 3 to smithi17031214-4 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: finishing write tid 4 to smithi17031214-4 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: oid 4 updating version 0 to 47 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: oid 4 updating version 47 to 48 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: oid 4 version 48 is already newer than 46 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 48 (ObjNum 505 snap 116 seq_num 505) dirty exists 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1034: left oid 4 (ObjNum 505 snap 116 seq_num 505) 2024-03-27T09:39:12.338 INFO:tasks.rados.rados.0.smithi170.stdout:1039: seq_num 508 ranges {537956=705790,1644029=772160} 2024-03-27T09:39:12.344 INFO:tasks.rados.rados.0.smithi170.stdout:1039: writing smithi17031214-21 from 537956 to 1243746 tid 1 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1039: writing smithi17031214-21 from 1644029 to 2416189 tid 2 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1034: done (3 left) 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1035: done (2 left) 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1040: write (excl) oid 49 current snap is 116 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1037: finishing write tid 1 to smithi17031214-9 2024-03-27T09:39:12.351 INFO:tasks.rados.rados.0.smithi170.stdout:1037: finishing write tid 2 to smithi17031214-9 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: finishing write tid 3 to smithi17031214-9 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: finishing write tid 4 to smithi17031214-9 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: oid 9 updating version 0 to 404 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: oid 9 version 404 is already newer than 404 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: oid 9 version 404 is already newer than 402 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 404 (ObjNum 506 snap 116 seq_num 506) dirty exists 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1037: left oid 9 (ObjNum 506 snap 116 seq_num 506) 2024-03-27T09:39:12.352 INFO:tasks.rados.rados.0.smithi170.stdout:1040: seq_num 509 ranges {554763=781039,2102511=452252} 2024-03-27T09:39:12.359 INFO:tasks.rados.rados.0.smithi170.stdout:1040: writing smithi17031214-49 from 554763 to 1335802 tid 1 2024-03-27T09:39:12.370 INFO:tasks.rados.rados.0.smithi170.stdout:1040: writing smithi17031214-49 from 2102511 to 2554763 tid 2 2024-03-27T09:39:12.370 INFO:tasks.rados.rados.0.smithi170.stdout:1037: done (2 left) 2024-03-27T09:39:12.370 INFO:tasks.rados.rados.0.smithi170.stdout:1041: read oid 2 snap 113 2024-03-27T09:39:12.370 INFO:tasks.rados.rados.0.smithi170.stdout:1041: expect deleted 2024-03-27T09:39:12.370 INFO:tasks.rados.rados.0.smithi170.stdout:1042: delete oid 8 current snap is 116 2024-03-27T09:39:12.382 INFO:tasks.rados.rados.0.smithi170.stdout:1039: finishing write tid 1 to smithi17031214-21 2024-03-27T09:39:12.384 INFO:tasks.rados.rados.0.smithi170.stdout:1040: finishing write tid 1 to smithi17031214-49 2024-03-27T09:39:12.410 INFO:tasks.rados.rados.0.smithi170.stdout:1040: finishing write tid 2 to smithi17031214-49 2024-03-27T09:39:12.421 INFO:tasks.rados.rados.0.smithi170.stdout:1039: finishing write tid 2 to smithi17031214-21 2024-03-27T09:39:12.424 INFO:tasks.rados.rados.0.smithi170.stdout:1040: finishing write tid 3 to smithi17031214-49 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:1040: finishing write tid 4 to smithi17031214-49 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:1040: oid 49 updating version 0 to 274 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:1040: oid 49 updating version 274 to 275 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:1040: oid 49 version 275 is already newer than 273 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 275 (ObjNum 509 snap 116 seq_num 509) dirty exists 2024-03-27T09:39:12.426 INFO:tasks.rados.rados.0.smithi170.stdout:1040: left oid 49 (ObjNum 509 snap 116 seq_num 509) 2024-03-27T09:39:12.427 INFO:tasks.rados.rados.0.smithi170.stdout:1040: done (3 left) 2024-03-27T09:39:12.427 INFO:tasks.rados.rados.0.smithi170.stdout:1041: done (2 left) 2024-03-27T09:39:12.427 INFO:tasks.rados.rados.0.smithi170.stdout:1042: done (1 left) 2024-03-27T09:39:12.427 INFO:tasks.rados.rados.0.smithi170.stdout:1043: snap_create 2024-03-27T09:39:12.434 INFO:tasks.rados.rados.0.smithi170.stdout:1039: finishing write tid 3 to smithi17031214-21 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:1039: finishing write tid 4 to smithi17031214-21 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:1039: oid 21 updating version 0 to 296 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:1039: oid 21 version 296 is already newer than 295 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:1039: oid 21 version 296 is already newer than 294 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 296 (ObjNum 508 snap 116 seq_num 508) dirty exists 2024-03-27T09:39:12.436 INFO:tasks.rados.rados.0.smithi170.stdout:1039: left oid 21 (ObjNum 508 snap 116 seq_num 508) 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1039: done (1 left) 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1043: done (0 left) 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1044: read oid 48 snap -1 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1044: expect (ObjNum 502 snap 116 seq_num 502) 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1045: write (excl) oid 45 current snap is 117 2024-03-27T09:39:13.021 INFO:tasks.rados.rados.0.smithi170.stdout:1045: seq_num 511 ranges {588377=647890,1935828=635840} 2024-03-27T09:39:13.027 INFO:tasks.rados.rados.0.smithi170.stdout:1045: writing smithi17031214-45 from 588377 to 1236267 tid 1 2024-03-27T09:39:13.036 INFO:tasks.rados.rados.0.smithi170.stdout:1045: writing smithi17031214-45 from 1935828 to 2571668 tid 2 2024-03-27T09:39:13.036 INFO:tasks.rados.rados.0.smithi170.stdout:1046: delete oid 9 current snap is 117 2024-03-27T09:39:13.089 INFO:tasks.rados.rados.0.smithi170.stdout:1044: expect (ObjNum 502 snap 116 seq_num 502) 2024-03-27T09:39:13.183 INFO:tasks.rados.rados.0.smithi170.stdout:1045: finishing write tid 1 to smithi17031214-45 2024-03-27T09:39:13.185 INFO:tasks.rados.rados.0.smithi170.stdout:1045: finishing write tid 2 to smithi17031214-45 2024-03-27T09:39:13.185 INFO:tasks.rados.rados.0.smithi170.stdout:1045: finishing write tid 3 to smithi17031214-45 2024-03-27T09:39:13.185 INFO:tasks.rados.rados.0.smithi170.stdout:1045: finishing write tid 4 to smithi17031214-45 2024-03-27T09:39:13.185 INFO:tasks.rados.rados.0.smithi170.stdout:1045: oid 45 updating version 0 to 352 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1045: oid 45 version 352 is already newer than 350 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1045: oid 45 version 352 is already newer than 352 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 352 (ObjNum 511 snap 117 seq_num 511) dirty exists 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1045: left oid 45 (ObjNum 511 snap 117 seq_num 511) 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1044: done (2 left) 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1045: done (1 left) 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1046: done (0 left) 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1047: read oid 22 snap -1 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1047: expect (ObjNum 426 snap 100 seq_num 426) 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1048: write oid 16 current snap is 117 2024-03-27T09:39:13.186 INFO:tasks.rados.rados.0.smithi170.stdout:1048: seq_num 513 ranges {621991=798388,1936439=639509} 2024-03-27T09:39:13.190 INFO:tasks.rados.rados.0.smithi170.stdout:1048: writing smithi17031214-16 from 621991 to 1420379 tid 1 2024-03-27T09:39:13.196 INFO:tasks.rados.rados.0.smithi170.stdout:1048: writing smithi17031214-16 from 1936439 to 2575948 tid 2 2024-03-27T09:39:13.196 INFO:tasks.rados.rados.0.smithi170.stdout:1049: read oid 7 snap 116 2024-03-27T09:39:13.196 INFO:tasks.rados.rados.0.smithi170.stdout:1049: expect deleted 2024-03-27T09:39:13.196 INFO:tasks.rados.rados.0.smithi170.stdout:1050: delete oid 10 current snap is 117 2024-03-27T09:39:13.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:13.209 INFO:tasks.rados.rados.0.smithi170.stdout:1047: expect (ObjNum 426 snap 100 seq_num 426) 2024-03-27T09:39:13.318 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:13.316+0000 7f932666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:13.322 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:13.321+0000 7f932666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:13.357 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:13.392 INFO:tasks.rados.rados.0.smithi170.stdout:1047: done (3 left) 2024-03-27T09:39:13.392 INFO:tasks.rados.rados.0.smithi170.stdout:1050: done (2 left) 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1051: snap_create 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: finishing write tid 1 to smithi17031214-16 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: finishing write tid 2 to smithi17031214-16 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: finishing write tid 3 to smithi17031214-16 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: finishing write tid 4 to smithi17031214-16 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: oid 16 updating version 0 to 106 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: oid 16 version 106 is already newer than 105 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: oid 16 version 106 is already newer than 104 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 106 (ObjNum 513 snap 117 seq_num 513) dirty exists 2024-03-27T09:39:13.393 INFO:tasks.rados.rados.0.smithi170.stdout:1048: left oid 16 (ObjNum 513 snap 117 seq_num 513) 2024-03-27T09:39:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:13.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:14.025 INFO:tasks.rados.rados.0.smithi170.stdout:1048: done (2 left) 2024-03-27T09:39:14.025 INFO:tasks.rados.rados.0.smithi170.stdout:1049: done (1 left) 2024-03-27T09:39:14.026 INFO:tasks.rados.rados.0.smithi170.stdout:1051: done (0 left) 2024-03-27T09:39:14.026 INFO:tasks.rados.rados.0.smithi170.stdout:1052: write (excl) oid 38 current snap is 118 2024-03-27T09:39:14.026 INFO:tasks.rados.rados.0.smithi170.stdout:1052: seq_num 515 ranges {655605=548886,1653403=643178} 2024-03-27T09:39:14.033 INFO:tasks.rados.rados.0.smithi170.stdout:1052: writing smithi17031214-38 from 655605 to 1204491 tid 1 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1052: writing smithi17031214-38 from 1653403 to 2296581 tid 2 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1053: read oid 25 snap -1 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1053: expect (ObjNum 499 snap 114 seq_num 499) 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1054: read oid 17 snap -1 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1054: expect deleted 2024-03-27T09:39:14.040 INFO:tasks.rados.rados.0.smithi170.stdout:1055: read oid 43 snap -1 2024-03-27T09:39:14.041 INFO:tasks.rados.rados.0.smithi170.stdout:1055: expect (ObjNum 486 snap 113 seq_num 486) 2024-03-27T09:39:14.041 INFO:tasks.rados.rados.0.smithi170.stdout:1056: read oid 18 snap 116 2024-03-27T09:39:14.041 INFO:tasks.rados.rados.0.smithi170.stdout:1056: expect deleted 2024-03-27T09:39:14.041 INFO:tasks.rados.rados.0.smithi170.stdout:1057: snap_create 2024-03-27T09:39:14.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~204s 2024-03-27T09:39:14.108 INFO:tasks.rados.rados.0.smithi170.stdout:1052: finishing write tid 1 to smithi17031214-38 2024-03-27T09:39:14.121 INFO:tasks.rados.rados.0.smithi170.stdout:1053: expect (ObjNum 499 snap 114 seq_num 499) 2024-03-27T09:39:14.278 INFO:tasks.rados.rados.0.smithi170.stdout:1052: finishing write tid 2 to smithi17031214-38 2024-03-27T09:39:14.278 INFO:tasks.rados.rados.0.smithi170.stdout:1055: expect (ObjNum 486 snap 113 seq_num 486) 2024-03-27T09:39:14.358 INFO:tasks.rados.rados.0.smithi170.stdout:1052: finishing write tid 3 to smithi17031214-38 2024-03-27T09:39:14.358 INFO:tasks.rados.rados.0.smithi170.stdout:1052: finishing write tid 4 to smithi17031214-38 2024-03-27T09:39:14.358 INFO:tasks.rados.rados.0.smithi170.stdout:1052: oid 38 updating version 0 to 298 2024-03-27T09:39:14.359 INFO:tasks.rados.rados.0.smithi170.stdout:1052: oid 38 updating version 298 to 299 2024-03-27T09:39:14.359 INFO:tasks.rados.rados.0.smithi170.stdout:1052: oid 38 version 299 is already newer than 297 2024-03-27T09:39:14.359 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 299 (ObjNum 515 snap 118 seq_num 515) dirty exists 2024-03-27T09:39:14.359 INFO:tasks.rados.rados.0.smithi170.stdout:1052: left oid 38 (ObjNum 515 snap 118 seq_num 515) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1052: done (5 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1053: done (4 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1054: done (3 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1055: done (2 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1056: done (1 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1057: done (0 left) 2024-03-27T09:39:15.030 INFO:tasks.rados.rados.0.smithi170.stdout:1058: snap_create 2024-03-27T09:39:15.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:15.514 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:15.512+0000 7f050e5cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:15.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:15.518 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:15.517+0000 7f050e5cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:15.543 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:15.589 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:15.618 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:15.625 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:15.977 INFO:tasks.rados.rados.0.smithi170.stdout:1058: done (0 left) 2024-03-27T09:39:15.977 INFO:tasks.rados.rados.0.smithi170.stdout:1059: read oid 24 snap 116 2024-03-27T09:39:15.978 INFO:tasks.rados.rados.0.smithi170.stdout:1059: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:39:15.978 INFO:tasks.rados.rados.0.smithi170.stdout:1060: snap_create 2024-03-27T09:39:16.031 INFO:tasks.rados.rados.0.smithi170.stdout:1059: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:39:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:1059: done (1 left) 2024-03-27T09:39:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:1060: done (0 left) 2024-03-27T09:39:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:1061: write (excl) oid 37 current snap is 121 2024-03-27T09:39:16.979 INFO:tasks.rados.rados.0.smithi170.stdout:1061: seq_num 516 ranges {672412=624135,1995532=676880} 2024-03-27T09:39:16.986 INFO:tasks.rados.rados.0.smithi170.stdout:1061: writing smithi17031214-37 from 672412 to 1296547 tid 1 2024-03-27T09:39:16.994 INFO:tasks.rados.rados.0.smithi170.stdout:1061: writing smithi17031214-37 from 1995532 to 2672412 tid 2 2024-03-27T09:39:16.994 INFO:tasks.rados.rados.0.smithi170.stdout:1062: read oid 22 snap -1 2024-03-27T09:39:16.994 INFO:tasks.rados.rados.0.smithi170.stdout:1062: expect (ObjNum 426 snap 100 seq_num 426) 2024-03-27T09:39:16.994 INFO:tasks.rados.rados.0.smithi170.stdout:1063: snap_remove snap 117 2024-03-27T09:39:17.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:17.704 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:17.702+0000 7ff8c66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:17.708 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:17.707+0000 7ff8c66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:17.732 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:17.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:17.979 INFO:tasks.rados.rados.0.smithi170.stdout:1063: done (2 left) 2024-03-27T09:39:17.979 INFO:tasks.rados.rados.0.smithi170.stdout:1064: read oid 15 snap -1 2024-03-27T09:39:17.979 INFO:tasks.rados.rados.0.smithi170.stdout:1064: expect (ObjNum 428 snap 100 seq_num 428) 2024-03-27T09:39:17.979 INFO:tasks.rados.rados.0.smithi170.stdout:1065: write (excl) oid 16 current snap is 121 2024-03-27T09:39:17.979 INFO:tasks.rados.rados.0.smithi170.stdout:1065: seq_num 517 ranges {689219=415737,1770367=646847} 2024-03-27T09:39:17.985 INFO:tasks.rados.rados.0.smithi170.stdout:1065: writing smithi17031214-16 from 689219 to 1104956 tid 1 2024-03-27T09:39:17.993 INFO:tasks.rados.rados.0.smithi170.stdout:1065: writing smithi17031214-16 from 1770367 to 2417214 tid 2 2024-03-27T09:39:17.994 INFO:tasks.rados.rados.0.smithi170.stdout:1066: read oid 6 snap -1 2024-03-27T09:39:17.994 INFO:tasks.rados.rados.0.smithi170.stdout:1066: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:17.994 INFO:tasks.rados.rados.0.smithi170.stdout:1067: write (excl) oid 30 current snap is 121 2024-03-27T09:39:17.994 INFO:tasks.rados.rados.0.smithi170.stdout:1067: seq_num 518 ranges {706026=490986,1828849=790505} 2024-03-27T09:39:18.000 INFO:tasks.rados.rados.0.smithi170.stdout:1067: writing smithi17031214-30 from 706026 to 1197012 tid 1 2024-03-27T09:39:18.012 INFO:tasks.rados.rados.0.smithi170.stdout:1067: writing smithi17031214-30 from 1828849 to 2619354 tid 2 2024-03-27T09:39:18.012 INFO:tasks.rados.rados.0.smithi170.stdout:1068: write oid 23 current snap is 121 2024-03-27T09:39:18.012 INFO:tasks.rados.rados.0.smithi170.stdout:1062: expect (ObjNum 426 snap 100 seq_num 426) 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: finishing write tid 1 to smithi17031214-37 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: finishing write tid 2 to smithi17031214-37 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: finishing write tid 3 to smithi17031214-37 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: finishing write tid 4 to smithi17031214-37 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: oid 37 updating version 0 to 351 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: oid 37 version 351 is already newer than 350 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: oid 37 updating version 351 to 352 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 352 (ObjNum 516 snap 121 seq_num 516) dirty exists 2024-03-27T09:39:18.184 INFO:tasks.rados.rados.0.smithi170.stdout:1061: left oid 37 (ObjNum 516 snap 121 seq_num 516) 2024-03-27T09:39:18.185 INFO:tasks.rados.rados.0.smithi170.stdout:1068: seq_num 519 ranges {722833=566235,1887331=650516} 2024-03-27T09:39:18.190 INFO:tasks.rados.rados.0.smithi170.stdout:1068: writing smithi17031214-23 from 722833 to 1289068 tid 1 2024-03-27T09:39:18.198 INFO:tasks.rados.rados.0.smithi170.stdout:1068: writing smithi17031214-23 from 1887331 to 2537847 tid 2 2024-03-27T09:39:18.198 INFO:tasks.rados.rados.0.smithi170.stdout:1061: done (6 left) 2024-03-27T09:39:18.198 INFO:tasks.rados.rados.0.smithi170.stdout:1062: done (5 left) 2024-03-27T09:39:18.198 INFO:tasks.rados.rados.0.smithi170.stdout:1069: write oid 18 current snap is 121 2024-03-27T09:39:18.198 INFO:tasks.rados.rados.0.smithi170.stdout:1069: seq_num 520 ranges {739640=641484,1829460=794174} 2024-03-27T09:39:18.205 INFO:tasks.rados.rados.0.smithi170.stdout:1069: writing smithi17031214-18 from 739640 to 1381124 tid 1 2024-03-27T09:39:18.215 INFO:tasks.rados.rados.0.smithi170.stdout:1069: writing smithi17031214-18 from 1829460 to 2623634 tid 2 2024-03-27T09:39:18.216 INFO:tasks.rados.rados.0.smithi170.stdout:1070: write oid 47 current snap is 121 2024-03-27T09:39:18.216 INFO:tasks.rados.rados.0.smithi170.stdout:1070: seq_num 521 ranges {756447=716733,1887942=537832} 2024-03-27T09:39:18.225 INFO:tasks.rados.rados.0.smithi170.stdout:1070: writing smithi17031214-47 from 756447 to 1473180 tid 1 2024-03-27T09:39:18.233 INFO:tasks.rados.rados.0.smithi170.stdout:1070: writing smithi17031214-47 from 1887942 to 2425774 tid 2 2024-03-27T09:39:18.233 INFO:tasks.rados.rados.0.smithi170.stdout:1071: snap_remove snap 119 2024-03-27T09:39:19.038 INFO:tasks.rados.rados.0.smithi170.stdout:1071: done (7 left) 2024-03-27T09:39:19.038 INFO:tasks.rados.rados.0.smithi170.stdout:1072: write oid 9 current snap is 121 2024-03-27T09:39:19.038 INFO:tasks.rados.rados.0.smithi170.stdout:1072: seq_num 522 ranges {773254=791982,2346424=426830} 2024-03-27T09:39:19.046 INFO:tasks.rados.rados.0.smithi170.stdout:1072: writing smithi17031214-9 from 773254 to 1565236 tid 1 2024-03-27T09:39:19.053 INFO:tasks.rados.rados.0.smithi170.stdout:1072: writing smithi17031214-9 from 2346424 to 2773254 tid 2 2024-03-27T09:39:19.053 INFO:tasks.rados.rados.0.smithi170.stdout:1073: write (excl) oid 36 current snap is 121 2024-03-27T09:39:19.053 INFO:tasks.rados.rados.0.smithi170.stdout:1073: seq_num 523 ranges {790061=467231,2004906=541501} 2024-03-27T09:39:19.061 INFO:tasks.rados.rados.0.smithi170.stdout:1073: writing smithi17031214-36 from 790061 to 1257292 tid 1 2024-03-27T09:39:19.068 INFO:tasks.rados.rados.0.smithi170.stdout:1073: writing smithi17031214-36 from 2004906 to 2546407 tid 2 2024-03-27T09:39:19.068 INFO:tasks.rados.rados.0.smithi170.stdout:1064: expect (ObjNum 428 snap 100 seq_num 428) 2024-03-27T09:39:19.275 INFO:tasks.rados.rados.0.smithi170.stdout:1066: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1067: finishing write tid 1 to smithi17031214-30 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1065: finishing write tid 1 to smithi17031214-16 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1067: finishing write tid 2 to smithi17031214-30 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1065: finishing write tid 2 to smithi17031214-16 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1065: finishing write tid 3 to smithi17031214-16 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1067: finishing write tid 3 to smithi17031214-30 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1067: finishing write tid 4 to smithi17031214-30 2024-03-27T09:39:19.401 INFO:tasks.rados.rados.0.smithi170.stdout:1067: oid 30 updating version 0 to 220 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1067: oid 30 updating version 220 to 222 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1067: oid 30 version 222 is already newer than 221 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 222 (ObjNum 518 snap 121 seq_num 518) dirty exists 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1067: left oid 30 (ObjNum 518 snap 121 seq_num 518) 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1065: finishing write tid 4 to smithi17031214-16 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1065: oid 16 updating version 0 to 110 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1065: oid 16 version 110 is already newer than 108 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1065: oid 16 version 110 is already newer than 109 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 110 (ObjNum 517 snap 121 seq_num 517) dirty exists 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1065: left oid 16 (ObjNum 517 snap 121 seq_num 517) 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1074: read oid 17 snap -1 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1074: expect deleted 2024-03-27T09:39:19.402 INFO:tasks.rados.rados.0.smithi170.stdout:1064: done (9 left) 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1065: done (8 left) 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1066: done (7 left) 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1067: done (6 left) 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1075: read oid 37 snap -1 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1075: expect (ObjNum 516 snap 121 seq_num 516) 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1076: delete oid 5 current snap is 121 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1068: finishing write tid 1 to smithi17031214-23 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1069: finishing write tid 1 to smithi17031214-18 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1068: finishing write tid 2 to smithi17031214-23 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1070: finishing write tid 1 to smithi17031214-47 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1069: finishing write tid 2 to smithi17031214-18 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1069: finishing write tid 3 to smithi17031214-18 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1070: finishing write tid 2 to smithi17031214-47 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1068: finishing write tid 3 to smithi17031214-23 2024-03-27T09:39:19.403 INFO:tasks.rados.rados.0.smithi170.stdout:1069: finishing write tid 4 to smithi17031214-18 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1069: oid 18 updating version 0 to 357 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1069: oid 18 version 357 is already newer than 355 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1069: oid 18 version 357 is already newer than 356 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 357 (ObjNum 520 snap 121 seq_num 520) dirty exists 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1069: left oid 18 (ObjNum 520 snap 121 seq_num 520) 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1068: finishing write tid 4 to smithi17031214-23 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1068: oid 23 updating version 0 to 410 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1068: oid 23 version 410 is already newer than 409 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1068: oid 23 version 410 is already newer than 408 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 410 (ObjNum 519 snap 121 seq_num 519) dirty exists 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1068: left oid 23 (ObjNum 519 snap 121 seq_num 519) 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1070: finishing write tid 3 to smithi17031214-47 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1070: finishing write tid 4 to smithi17031214-47 2024-03-27T09:39:19.404 INFO:tasks.rados.rados.0.smithi170.stdout:1070: oid 47 updating version 0 to 171 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1070: oid 47 updating version 171 to 172 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1070: oid 47 version 172 is already newer than 170 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 172 (ObjNum 521 snap 121 seq_num 521) dirty exists 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1070: left oid 47 (ObjNum 521 snap 121 seq_num 521) 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: finishing write tid 1 to smithi17031214-9 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: finishing write tid 2 to smithi17031214-9 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1073: finishing write tid 1 to smithi17031214-36 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: finishing write tid 3 to smithi17031214-9 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: finishing write tid 4 to smithi17031214-9 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: oid 9 updating version 0 to 413 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: oid 9 updating version 413 to 415 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: oid 9 version 415 is already newer than 415 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 415 (ObjNum 522 snap 121 seq_num 522) dirty exists 2024-03-27T09:39:19.405 INFO:tasks.rados.rados.0.smithi170.stdout:1072: left oid 9 (ObjNum 522 snap 121 seq_num 522) 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: finishing write tid 2 to smithi17031214-36 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: finishing write tid 3 to smithi17031214-36 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: finishing write tid 4 to smithi17031214-36 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: oid 36 updating version 0 to 361 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: oid 36 version 361 is already newer than 359 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: oid 36 version 361 is already newer than 361 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 361 (ObjNum 523 snap 121 seq_num 523) dirty exists 2024-03-27T09:39:19.406 INFO:tasks.rados.rados.0.smithi170.stdout:1073: left oid 36 (ObjNum 523 snap 121 seq_num 523) 2024-03-27T09:39:19.427 INFO:tasks.rados.rados.0.smithi170.stdout:1075: expect (ObjNum 516 snap 121 seq_num 516) 2024-03-27T09:39:19.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~209s 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1068: done (7 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1069: done (6 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1070: done (5 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1072: done (4 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1073: done (3 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1074: done (2 left) 2024-03-27T09:39:19.594 INFO:tasks.rados.rados.0.smithi170.stdout:1075: done (1 left) 2024-03-27T09:39:19.595 INFO:tasks.rados.rados.0.smithi170.stdout:1076: done (0 left) 2024-03-27T09:39:19.595 INFO:tasks.rados.rados.0.smithi170.stdout:1077: snap_create 2024-03-27T09:39:19.783 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:19.891 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:19.889+0000 7f54c9c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:19.894 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:19.894+0000 7f54c9c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:19.919 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:19.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:20.042 INFO:tasks.rados.rados.0.smithi170.stdout:1077: done (0 left) 2024-03-27T09:39:20.042 INFO:tasks.rados.rados.0.smithi170.stdout:1078: delete oid 17 current snap is 122 2024-03-27T09:39:20.095 INFO:tasks.rados.rados.0.smithi170.stdout:1078: done (0 left) 2024-03-27T09:39:20.096 INFO:tasks.rados.rados.0.smithi170.stdout:1079: write oid 1 current snap is 122 2024-03-27T09:39:20.096 INFO:tasks.rados.rados.0.smithi170.stdout:1079: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-03-27T09:39:20.100 INFO:tasks.rados.rados.0.smithi170.stdout:1079: writing smithi17031214-1 from 440482 to 849813 tid 1 2024-03-27T09:39:20.105 INFO:tasks.rados.rados.0.smithi170.stdout:1079: writing smithi17031214-1 from 1380352 to 1785533 tid 2 2024-03-27T09:39:20.111 INFO:tasks.rados.rados.0.smithi170.stdout:1079: writing smithi17031214-1 from 2461553 to 2840482 tid 3 2024-03-27T09:39:20.111 INFO:tasks.rados.rados.0.smithi170.stdout:1080: snap_remove snap 121 2024-03-27T09:39:20.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:20.726 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:20.729 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:21.045 INFO:tasks.rados.rados.0.smithi170.stdout:1080: done (1 left) 2024-03-27T09:39:21.045 INFO:tasks.rados.rados.0.smithi170.stdout:1081: read oid 7 snap 113 2024-03-27T09:39:21.045 INFO:tasks.rados.rados.0.smithi170.stdout:1081: expect deleted 2024-03-27T09:39:21.045 INFO:tasks.rados.rados.0.smithi170.stdout:1082: write (excl) oid 9 current snap is 122 2024-03-27T09:39:21.045 INFO:tasks.rados.rados.0.smithi170.stdout:1079: finishing write tid 1 to smithi17031214-1 2024-03-27T09:39:21.046 INFO:tasks.rados.rados.0.smithi170.stdout:1082: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-03-27T09:39:21.050 INFO:tasks.rados.rados.0.smithi170.stdout:1082: writing smithi17031214-9 from 457289 to 941869 tid 1 2024-03-27T09:39:21.056 INFO:tasks.rados.rados.0.smithi170.stdout:1082: writing smithi17031214-9 from 1438834 to 1987673 tid 2 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1082: writing smithi17031214-9 from 2772623 to 2857289 tid 3 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1083: read oid 47 snap -1 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1083: expect (ObjNum 521 snap 121 seq_num 521) 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1084: read oid 13 snap -1 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1084: expect (ObjNum 466 snap 110 seq_num 466) 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1085: read oid 20 snap -1 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1085: expect (ObjNum 477 snap 112 seq_num 477) 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1086: read oid 14 snap -1 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1086: expect (ObjNum 497 snap 113 seq_num 497) 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1087: write oid 8 current snap is 122 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1079: finishing write tid 2 to smithi17031214-1 2024-03-27T09:39:21.062 INFO:tasks.rados.rados.0.smithi170.stdout:1087: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-03-27T09:39:21.067 INFO:tasks.rados.rados.0.smithi170.stdout:1087: writing smithi17031214-8 from 474096 to 1033925 tid 1 2024-03-27T09:39:21.075 INFO:tasks.rados.rados.0.smithi170.stdout:1087: writing smithi17031214-8 from 1780963 to 2189813 tid 2 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1087: writing smithi17031214-8 from 2800046 to 2874096 tid 3 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1088: read oid 32 snap -1 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1088: expect (ObjNum 312 snap 81 seq_num 312) 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1089: read oid 17 snap -1 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1089: expect deleted 2024-03-27T09:39:21.080 INFO:tasks.rados.rados.0.smithi170.stdout:1090: snap_remove snap 118 2024-03-27T09:39:21.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:22.052 INFO:tasks.rados.rados.0.smithi170.stdout:1090: done (10 left) 2024-03-27T09:39:22.052 INFO:tasks.rados.rados.0.smithi170.stdout:1091: read oid 27 snap -1 2024-03-27T09:39:22.052 INFO:tasks.rados.rados.0.smithi170.stdout:1091: expect (ObjNum 490 snap 113 seq_num 490) 2024-03-27T09:39:22.053 INFO:tasks.rados.rados.0.smithi170.stdout:1092: read oid 40 snap -1 2024-03-27T09:39:22.053 INFO:tasks.rados.rados.0.smithi170.stdout:1092: expect deleted 2024-03-27T09:39:22.053 INFO:tasks.rados.rados.0.smithi170.stdout:1093: write oid 22 current snap is 122 2024-03-27T09:39:22.053 INFO:tasks.rados.rados.0.smithi170.stdout:1079: finishing write tid 3 to smithi17031214-1 2024-03-27T09:39:22.053 INFO:tasks.rados.rados.0.smithi170.stdout:1093: seq_num 529 ranges {490903=635078,1839445=552508} 2024-03-27T09:39:22.058 INFO:tasks.rados.rados.0.smithi170.stdout:1093: writing smithi17031214-22 from 490903 to 1125981 tid 1 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1093: writing smithi17031214-22 from 1839445 to 2391953 tid 2 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1094: write oid 34 current snap is 122 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1079: finishing write tid 4 to smithi17031214-1 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1079: finishing write tid 5 to smithi17031214-1 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1079: oid 1 updating version 0 to 184 2024-03-27T09:39:22.067 INFO:tasks.rados.rados.0.smithi170.stdout:1079: oid 1 version 184 is already newer than 183 2024-03-27T09:39:22.068 INFO:tasks.rados.rados.0.smithi170.stdout:1079: oid 1 updating version 184 to 185 2024-03-27T09:39:22.068 INFO:tasks.rados.rados.0.smithi170.stdout:1079: oid 1 updating version 185 to 186 2024-03-27T09:39:22.068 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 186 (ObjNum 526 snap 122 seq_num 526) dirty exists 2024-03-27T09:39:22.068 INFO:tasks.rados.rados.0.smithi170.stdout:1079: left oid 1 (ObjNum 526 snap 122 seq_num 526) 2024-03-27T09:39:22.068 INFO:tasks.rados.rados.0.smithi170.stdout:1094: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-03-27T09:39:22.073 INFO:tasks.rados.rados.0.smithi170.stdout:1094: writing smithi17031214-34 from 507710 to 1218037 tid 1 2024-03-27T09:39:22.079 INFO:tasks.rados.rados.0.smithi170.stdout:1094: writing smithi17031214-34 from 1897927 to 2310446 tid 2 2024-03-27T09:39:22.083 INFO:tasks.rados.rados.0.smithi170.stdout:1094: writing smithi17031214-34 from 2854892 to 2907710 tid 3 2024-03-27T09:39:22.083 INFO:tasks.rados.rados.0.smithi170.stdout:1079: done (13 left) 2024-03-27T09:39:22.083 INFO:tasks.rados.rados.0.smithi170.stdout:1095: write (excl) oid 43 current snap is 122 2024-03-27T09:39:22.083 INFO:tasks.rados.rados.0.smithi170.stdout:1095: seq_num 531 ranges {524517=785576,1956409=556177} 2024-03-27T09:39:22.093 INFO:tasks.rados.rados.0.smithi170.stdout:1095: writing smithi17031214-43 from 524517 to 1310093 tid 1 2024-03-27T09:39:22.103 INFO:tasks.rados.rados.0.smithi170.stdout:1095: writing smithi17031214-43 from 1956409 to 2512586 tid 2 2024-03-27T09:39:22.103 INFO:tasks.rados.rados.0.smithi170.stdout:1096: snap_create 2024-03-27T09:39:22.103 INFO:tasks.rados.rados.0.smithi170.stdout:1082: finishing write tid 1 to smithi17031214-9 2024-03-27T09:39:22.103 INFO:tasks.rados.rados.0.smithi170.stdout:1088: expect (ObjNum 312 snap 81 seq_num 312) 2024-03-27T09:39:22.105 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:22.103+0000 7ff911e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:22.111 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:22.110+0000 7ff911e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:22.172 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:22.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:22.333 INFO:tasks.rados.rados.0.smithi170.stdout:1083: expect (ObjNum 521 snap 121 seq_num 521) 2024-03-27T09:39:22.435 INFO:tasks.rados.rados.0.smithi170.stdout:1084: expect (ObjNum 466 snap 110 seq_num 466) 2024-03-27T09:39:22.576 INFO:tasks.rados.rados.0.smithi170.stdout:1085: expect (ObjNum 477 snap 112 seq_num 477) 2024-03-27T09:39:22.652 INFO:tasks.rados.rados.0.smithi170.stdout:1082: finishing write tid 2 to smithi17031214-9 2024-03-27T09:39:22.652 INFO:tasks.rados.rados.0.smithi170.stdout:1086: expect (ObjNum 497 snap 113 seq_num 497) 2024-03-27T09:39:22.740 INFO:tasks.rados.rados.0.smithi170.stdout:1082: finishing write tid 3 to smithi17031214-9 2024-03-27T09:39:22.740 INFO:tasks.rados.rados.0.smithi170.stdout:1087: finishing write tid 1 to smithi17031214-8 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: finishing write tid 4 to smithi17031214-9 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: finishing write tid 5 to smithi17031214-9 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: oid 9 updating version 0 to 419 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: oid 9 version 419 is already newer than 416 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: oid 9 version 419 is already newer than 418 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: oid 9 version 419 is already newer than 418 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 419 (ObjNum 527 snap 122 seq_num 527) dirty exists 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1082: left oid 9 (ObjNum 527 snap 122 seq_num 527) 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: finishing write tid 2 to smithi17031214-8 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: finishing write tid 3 to smithi17031214-8 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: finishing write tid 4 to smithi17031214-8 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: finishing write tid 5 to smithi17031214-8 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: oid 8 updating version 0 to 364 2024-03-27T09:39:22.741 INFO:tasks.rados.rados.0.smithi170.stdout:1087: oid 8 updating version 364 to 366 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1087: oid 8 version 366 is already newer than 363 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1087: oid 8 version 366 is already newer than 365 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 366 (ObjNum 528 snap 122 seq_num 528) dirty exists 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1087: left oid 8 (ObjNum 528 snap 122 seq_num 528) 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1093: finishing write tid 1 to smithi17031214-22 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1094: finishing write tid 1 to smithi17031214-34 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1093: finishing write tid 2 to smithi17031214-22 2024-03-27T09:39:22.742 INFO:tasks.rados.rados.0.smithi170.stdout:1091: expect (ObjNum 490 snap 113 seq_num 490) 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1094: finishing write tid 2 to smithi17031214-34 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: finishing write tid 3 to smithi17031214-22 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1095: finishing write tid 1 to smithi17031214-43 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: finishing write tid 4 to smithi17031214-22 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: oid 22 updating version 0 to 188 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: oid 22 updating version 188 to 189 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: oid 22 updating version 189 to 190 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 190 (ObjNum 529 snap 122 seq_num 529) dirty exists 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1093: left oid 22 (ObjNum 529 snap 122 seq_num 529) 2024-03-27T09:39:22.886 INFO:tasks.rados.rados.0.smithi170.stdout:1094: finishing write tid 3 to smithi17031214-34 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: finishing write tid 4 to smithi17031214-34 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: finishing write tid 5 to smithi17031214-34 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: oid 34 updating version 0 to 424 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: oid 34 updating version 424 to 425 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: oid 34 version 425 is already newer than 423 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: oid 34 updating version 425 to 426 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 426 (ObjNum 530 snap 122 seq_num 530) dirty exists 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1094: left oid 34 (ObjNum 530 snap 122 seq_num 530) 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: finishing write tid 2 to smithi17031214-43 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: finishing write tid 3 to smithi17031214-43 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: finishing write tid 4 to smithi17031214-43 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: oid 43 updating version 0 to 355 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: oid 43 version 355 is already newer than 354 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:1095: oid 43 updating version 355 to 356 2024-03-27T09:39:22.887 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 356 (ObjNum 531 snap 122 seq_num 531) dirty exists 2024-03-27T09:39:22.888 INFO:tasks.rados.rados.0.smithi170.stdout:1095: left oid 43 (ObjNum 531 snap 122 seq_num 531) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1081: done (14 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1082: done (13 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1083: done (12 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1084: done (11 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1085: done (10 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1086: done (9 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1087: done (8 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1088: done (7 left) 2024-03-27T09:39:23.053 INFO:tasks.rados.rados.0.smithi170.stdout:1089: done (6 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1091: done (5 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1092: done (4 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1093: done (3 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1094: done (2 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1095: done (1 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1096: done (0 left) 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1097: write oid 2 current snap is 123 2024-03-27T09:39:23.054 INFO:tasks.rados.rados.0.smithi170.stdout:1097: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-03-27T09:39:23.057 INFO:tasks.rados.rados.0.smithi170.stdout:1097: writing smithi17031214-2 from 541324 to 1002149 tid 1 2024-03-27T09:39:23.062 INFO:tasks.rados.rados.0.smithi170.stdout:1097: writing smithi17031214-2 from 1498538 to 1914726 tid 2 2024-03-27T09:39:23.069 INFO:tasks.rados.rados.0.smithi170.stdout:1097: writing smithi17031214-2 from 2393385 to 2941324 tid 3 2024-03-27T09:39:23.069 INFO:tasks.rados.rados.0.smithi170.stdout:1098: read oid 6 snap -1 2024-03-27T09:39:23.069 INFO:tasks.rados.rados.0.smithi170.stdout:1098: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:23.069 INFO:tasks.rados.rados.0.smithi170.stdout:1099: delete oid 13 current snap is 123 2024-03-27T09:39:23.115 INFO:tasks.rados.rados.0.smithi170.stdout:1098: expect (ObjNum 433 snap 100 seq_num 433) 2024-03-27T09:39:23.254 INFO:tasks.rados.rados.0.smithi170.stdout:1097: finishing write tid 1 to smithi17031214-2 2024-03-27T09:39:23.254 INFO:tasks.rados.rados.0.smithi170.stdout:1097: finishing write tid 2 to smithi17031214-2 2024-03-27T09:39:23.254 INFO:tasks.rados.rados.0.smithi170.stdout:1097: finishing write tid 3 to smithi17031214-2 2024-03-27T09:39:23.254 INFO:tasks.rados.rados.0.smithi170.stdout:1097: finishing write tid 4 to smithi17031214-2 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: finishing write tid 5 to smithi17031214-2 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: oid 2 updating version 0 to 360 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: oid 2 version 360 is already newer than 358 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: oid 2 version 360 is already newer than 357 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: oid 2 version 360 is already newer than 359 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 360 (ObjNum 532 snap 123 seq_num 532) dirty exists 2024-03-27T09:39:23.255 INFO:tasks.rados.rados.0.smithi170.stdout:1097: left oid 2 (ObjNum 532 snap 123 seq_num 532) 2024-03-27T09:39:23.256 INFO:tasks.rados.rados.0.smithi170.stdout:1097: done (2 left) 2024-03-27T09:39:23.256 INFO:tasks.rados.rados.0.smithi170.stdout:1098: done (1 left) 2024-03-27T09:39:23.256 INFO:tasks.rados.rados.0.smithi170.stdout:1099: done (0 left) 2024-03-27T09:39:23.256 INFO:tasks.rados.rados.0.smithi170.stdout:1100: snap_remove snap 122 2024-03-27T09:39:24.056 INFO:tasks.rados.rados.0.smithi170.stdout:1100: done (0 left) 2024-03-27T09:39:24.056 INFO:tasks.rados.rados.0.smithi170.stdout:1101: snap_remove snap 116 2024-03-27T09:39:24.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:24.429 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:24.427+0000 7f66d3f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:24.433 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:24.432+0000 7f66d3f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:24.457 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:24.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:25.060 INFO:tasks.rados.rados.0.smithi170.stdout:1101: done (0 left) 2024-03-27T09:39:25.061 INFO:tasks.rados.rados.0.smithi170.stdout:1102: snap_create 2024-03-27T09:39:25.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2024-03-27T09:39:25.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:25.844 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:25.848 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:25.979 INFO:tasks.rados.rados.0.smithi170.stdout:1102: done (0 left) 2024-03-27T09:39:25.979 INFO:tasks.rados.rados.0.smithi170.stdout:1103: write (excl) oid 40 current snap is 124 2024-03-27T09:39:25.980 INFO:tasks.rados.rados.0.smithi170.stdout:1103: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-03-27T09:39:25.987 INFO:tasks.rados.rados.0.smithi170.stdout:1103: writing smithi17031214-40 from 574938 to 1302614 tid 1 2024-03-27T09:39:25.994 INFO:tasks.rados.rados.0.smithi170.stdout:1103: writing smithi17031214-40 from 1731855 to 2435359 tid 2 2024-03-27T09:39:25.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: writing smithi17031214-40 from 2848231 to 2974938 tid 3 2024-03-27T09:39:25.998 INFO:tasks.rados.rados.0.smithi170.stdout:1104: write (excl) oid 48 current snap is 124 2024-03-27T09:39:25.998 INFO:tasks.rados.rados.0.smithi170.stdout:1104: seq_num 535 ranges {591745=402925,1790337=563515} 2024-03-27T09:39:26.005 INFO:tasks.rados.rados.0.smithi170.stdout:1104: writing smithi17031214-48 from 591745 to 994670 tid 1 2024-03-27T09:39:26.015 INFO:tasks.rados.rados.0.smithi170.stdout:1104: writing smithi17031214-48 from 1790337 to 2353852 tid 2 2024-03-27T09:39:26.016 INFO:tasks.rados.rados.0.smithi170.stdout:1105: write (excl) oid 13 current snap is 124 2024-03-27T09:39:26.016 INFO:tasks.rados.rados.0.smithi170.stdout:1105: seq_num 536 ranges {608552=478174,1848819=707173} 2024-03-27T09:39:26.023 INFO:tasks.rados.rados.0.smithi170.stdout:1105: writing smithi17031214-13 from 608552 to 1086726 tid 1 2024-03-27T09:39:26.034 INFO:tasks.rados.rados.0.smithi170.stdout:1105: writing smithi17031214-13 from 1848819 to 2555992 tid 2 2024-03-27T09:39:26.035 INFO:tasks.rados.rados.0.smithi170.stdout:1106: write (excl) oid 43 current snap is 124 2024-03-27T09:39:26.035 INFO:tasks.rados.rados.0.smithi170.stdout:1106: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-03-27T09:39:26.043 INFO:tasks.rados.rados.0.smithi170.stdout:1106: writing smithi17031214-43 from 625359 to 1178782 tid 1 2024-03-27T09:39:26.052 INFO:tasks.rados.rados.0.smithi170.stdout:1106: writing smithi17031214-43 from 1790948 to 2358132 tid 2 2024-03-27T09:39:26.057 INFO:tasks.rados.rados.0.smithi170.stdout:1106: writing smithi17031214-43 from 2930500 to 3025359 tid 3 2024-03-27T09:39:26.057 INFO:tasks.rados.rados.0.smithi170.stdout:1107: snap_remove snap 120 2024-03-27T09:39:26.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:26.619 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:26.616+0000 7f2107e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:26.622 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:26.622+0000 7f2107e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:26.646 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:26.694 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:26.981 INFO:tasks.rados.rados.0.smithi170.stdout:1107: done (4 left) 2024-03-27T09:39:26.981 INFO:tasks.rados.rados.0.smithi170.stdout:1108: write oid 3 current snap is 124 2024-03-27T09:39:26.981 INFO:tasks.rados.rados.0.smithi170.stdout:1103: finishing write tid 1 to smithi17031214-40 2024-03-27T09:39:26.981 INFO:tasks.rados.rados.0.smithi170.stdout:1108: seq_num 538 ranges {642166=628672,1849430=710842} 2024-03-27T09:39:26.987 INFO:tasks.rados.rados.0.smithi170.stdout:1108: writing smithi17031214-3 from 642166 to 1270838 tid 1 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1108: writing smithi17031214-3 from 1849430 to 2560272 tid 2 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1109: snap_create 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1104: finishing write tid 1 to smithi17031214-48 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1105: finishing write tid 1 to smithi17031214-13 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1104: finishing write tid 2 to smithi17031214-48 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1103: finishing write tid 2 to smithi17031214-40 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1104: finishing write tid 3 to smithi17031214-48 2024-03-27T09:39:26.996 INFO:tasks.rados.rados.0.smithi170.stdout:1104: finishing write tid 4 to smithi17031214-48 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1104: oid 48 updating version 0 to 279 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1104: oid 48 version 279 is already newer than 277 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1104: oid 48 version 279 is already newer than 278 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 279 (ObjNum 535 snap 124 seq_num 535) dirty exists 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1104: left oid 48 (ObjNum 535 snap 124 seq_num 535) 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: finishing write tid 2 to smithi17031214-13 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1103: finishing write tid 3 to smithi17031214-40 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: finishing write tid 3 to smithi17031214-13 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: finishing write tid 4 to smithi17031214-13 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: oid 13 updating version 0 to 234 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: oid 13 version 234 is already newer than 233 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: oid 13 updating version 234 to 235 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 235 (ObjNum 536 snap 124 seq_num 536) dirty exists 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1105: left oid 13 (ObjNum 536 snap 124 seq_num 536) 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1106: finishing write tid 1 to smithi17031214-43 2024-03-27T09:39:26.997 INFO:tasks.rados.rados.0.smithi170.stdout:1103: finishing write tid 4 to smithi17031214-40 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: finishing write tid 5 to smithi17031214-40 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: oid 40 updating version 0 to 433 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: oid 40 updating version 433 to 434 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: oid 40 version 434 is already newer than 431 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: oid 40 version 434 is already newer than 432 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 434 (ObjNum 534 snap 124 seq_num 534) dirty exists 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1103: left oid 40 (ObjNum 534 snap 124 seq_num 534) 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1106: finishing write tid 2 to smithi17031214-43 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1106: finishing write tid 3 to smithi17031214-43 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1106: finishing write tid 4 to smithi17031214-43 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1106: finishing write tid 5 to smithi17031214-43 2024-03-27T09:39:26.998 INFO:tasks.rados.rados.0.smithi170.stdout:1106: oid 43 updating version 0 to 364 2024-03-27T09:39:26.999 INFO:tasks.rados.rados.0.smithi170.stdout:1106: oid 43 updating version 364 to 365 2024-03-27T09:39:26.999 INFO:tasks.rados.rados.0.smithi170.stdout:1106: oid 43 updating version 365 to 366 2024-03-27T09:39:26.999 INFO:tasks.rados.rados.0.smithi170.stdout:1106: oid 43 updating version 366 to 367 2024-03-27T09:39:26.999 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 367 (ObjNum 537 snap 124 seq_num 537) dirty exists 2024-03-27T09:39:26.999 INFO:tasks.rados.rados.0.smithi170.stdout:1106: left oid 43 (ObjNum 537 snap 124 seq_num 537) 2024-03-27T09:39:27.079 INFO:tasks.rados.rados.0.smithi170.stdout:1108: finishing write tid 1 to smithi17031214-3 2024-03-27T09:39:27.106 INFO:tasks.rados.rados.0.smithi170.stdout:1108: finishing write tid 2 to smithi17031214-3 2024-03-27T09:39:27.112 INFO:tasks.rados.rados.0.smithi170.stdout:1108: finishing write tid 3 to smithi17031214-3 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:1108: finishing write tid 4 to smithi17031214-3 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:1108: oid 3 updating version 0 to 283 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:1108: oid 3 updating version 283 to 285 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:1108: oid 3 version 285 is already newer than 284 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 285 (ObjNum 538 snap 124 seq_num 538) dirty exists 2024-03-27T09:39:27.115 INFO:tasks.rados.rados.0.smithi170.stdout:1108: left oid 3 (ObjNum 538 snap 124 seq_num 538) 2024-03-27T09:39:27.981 INFO:tasks.rados.rados.0.smithi170.stdout:1103: done (5 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1104: done (4 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1105: done (3 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1106: done (2 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1108: done (1 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1109: done (0 left) 2024-03-27T09:39:27.982 INFO:tasks.rados.rados.0.smithi170.stdout:1110: snap_create 2024-03-27T09:39:28.695 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:28.817 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:28.815+0000 7f7d8a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:28.821 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:28.820+0000 7f7d8a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:28.845 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:28.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:28.987 INFO:tasks.rados.rados.0.smithi170.stdout:1110: done (0 left) 2024-03-27T09:39:28.987 INFO:tasks.rados.rados.0.smithi170.stdout:1111: read oid 17 snap -1 2024-03-27T09:39:28.987 INFO:tasks.rados.rados.0.smithi170.stdout:1111: expect deleted 2024-03-27T09:39:28.987 INFO:tasks.rados.rados.0.smithi170.stdout:1112: write oid 12 current snap is 126 2024-03-27T09:39:28.987 INFO:tasks.rados.rados.0.smithi170.stdout:1112: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-03-27T09:39:28.995 INFO:tasks.rados.rados.0.smithi170.stdout:1112: writing smithi17031214-12 from 658973 to 1362894 tid 1 2024-03-27T09:39:29.001 INFO:tasks.rados.rados.0.smithi170.stdout:1112: writing smithi17031214-12 from 1907912 to 2478765 tid 2 2024-03-27T09:39:29.005 INFO:tasks.rados.rados.0.smithi170.stdout:1112: writing smithi17031214-12 from 2985346 to 3058973 tid 3 2024-03-27T09:39:29.005 INFO:tasks.rados.rados.0.smithi170.stdout:1113: read oid 7 snap -1 2024-03-27T09:39:29.005 INFO:tasks.rados.rados.0.smithi170.stdout:1113: expect deleted 2024-03-27T09:39:29.005 INFO:tasks.rados.rados.0.smithi170.stdout:1114: snap_remove snap 124 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1114: done (3 left) 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1115: snap_create 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: finishing write tid 1 to smithi17031214-12 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: finishing write tid 2 to smithi17031214-12 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: finishing write tid 3 to smithi17031214-12 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: finishing write tid 4 to smithi17031214-12 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: finishing write tid 5 to smithi17031214-12 2024-03-27T09:39:29.989 INFO:tasks.rados.rados.0.smithi170.stdout:1112: oid 12 updating version 0 to 327 2024-03-27T09:39:29.990 INFO:tasks.rados.rados.0.smithi170.stdout:1112: oid 12 version 327 is already newer than 325 2024-03-27T09:39:29.990 INFO:tasks.rados.rados.0.smithi170.stdout:1112: oid 12 version 327 is already newer than 327 2024-03-27T09:39:29.990 INFO:tasks.rados.rados.0.smithi170.stdout:1112: oid 12 updating version 327 to 328 2024-03-27T09:39:29.990 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 328 (ObjNum 539 snap 126 seq_num 539) dirty exists 2024-03-27T09:39:29.990 INFO:tasks.rados.rados.0.smithi170.stdout:1112: left oid 12 (ObjNum 539 snap 126 seq_num 539) 2024-03-27T09:39:30.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2024-03-27T09:39:30.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:30.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:30.959 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:30.962 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1111: done (3 left) 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1112: done (2 left) 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1113: done (1 left) 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1115: done (0 left) 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1116: write oid 1 current snap is 127 2024-03-27T09:39:30.994 INFO:tasks.rados.rados.0.smithi170.stdout:1116: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-03-27T09:39:30.997 INFO:tasks.rados.rados.0.smithi170.stdout:1116: writing smithi17031214-1 from 675780 to 1171303 tid 1 2024-03-27T09:39:31.004 INFO:tasks.rados.rados.0.smithi170.stdout:1116: writing smithi17031214-1 from 1682747 to 2397258 tid 2 2024-03-27T09:39:31.008 INFO:tasks.rados.rados.0.smithi170.stdout:1116: writing smithi17031214-1 from 3012769 to 3075780 tid 3 2024-03-27T09:39:31.008 INFO:tasks.rados.rados.0.smithi170.stdout:1117: delete oid 30 current snap is 127 2024-03-27T09:39:31.014 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:31.011+0000 7f4766260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:31.018 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:31.017+0000 7f4766260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:31.044 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:31.087 INFO:tasks.rados.rados.0.smithi170.stdout:1117: done (1 left) 2024-03-27T09:39:31.087 INFO:tasks.rados.rados.0.smithi170.stdout:1118: read oid 24 snap -1 2024-03-27T09:39:31.087 INFO:tasks.rados.rados.0.smithi170.stdout:1118: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:39:31.087 INFO:tasks.rados.rados.0.smithi170.stdout:1119: snap_remove snap 125 2024-03-27T09:39:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:31.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:31.998 INFO:tasks.rados.rados.0.smithi170.stdout:1119: done (2 left) 2024-03-27T09:39:31.998 INFO:tasks.rados.rados.0.smithi170.stdout:1120: snap_create 2024-03-27T09:39:31.998 INFO:tasks.rados.rados.0.smithi170.stdout:1116: finishing write tid 1 to smithi17031214-1 2024-03-27T09:39:31.998 INFO:tasks.rados.rados.0.smithi170.stdout:1116: finishing write tid 2 to smithi17031214-1 2024-03-27T09:39:31.998 INFO:tasks.rados.rados.0.smithi170.stdout:1118: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:39:32.145 INFO:tasks.rados.rados.0.smithi170.stdout:1116: finishing write tid 3 to smithi17031214-1 2024-03-27T09:39:32.145 INFO:tasks.rados.rados.0.smithi170.stdout:1116: finishing write tid 4 to smithi17031214-1 2024-03-27T09:39:32.145 INFO:tasks.rados.rados.0.smithi170.stdout:1116: finishing write tid 5 to smithi17031214-1 2024-03-27T09:39:32.145 INFO:tasks.rados.rados.0.smithi170.stdout:1116: oid 1 updating version 0 to 199 2024-03-27T09:39:32.145 INFO:tasks.rados.rados.0.smithi170.stdout:1116: oid 1 version 199 is already newer than 198 2024-03-27T09:39:32.146 INFO:tasks.rados.rados.0.smithi170.stdout:1116: oid 1 version 199 is already newer than 197 2024-03-27T09:39:32.146 INFO:tasks.rados.rados.0.smithi170.stdout:1116: oid 1 version 199 is already newer than 197 2024-03-27T09:39:32.146 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 199 (ObjNum 540 snap 127 seq_num 540) dirty exists 2024-03-27T09:39:32.146 INFO:tasks.rados.rados.0.smithi170.stdout:1116: left oid 1 (ObjNum 540 snap 127 seq_num 540) 2024-03-27T09:39:33.001 INFO:tasks.rados.rados.0.smithi170.stdout:1116: done (2 left) 2024-03-27T09:39:33.001 INFO:tasks.rados.rados.0.smithi170.stdout:1118: done (1 left) 2024-03-27T09:39:33.001 INFO:tasks.rados.rados.0.smithi170.stdout:1120: done (0 left) 2024-03-27T09:39:33.001 INFO:tasks.rados.rados.0.smithi170.stdout:1121: write oid 48 current snap is 128 2024-03-27T09:39:33.002 INFO:tasks.rados.rados.0.smithi170.stdout:1121: seq_num 542 ranges {709394=646021,2083358=718180} 2024-03-27T09:39:33.008 INFO:tasks.rados.rados.0.smithi170.stdout:1121: writing smithi17031214-48 from 709394 to 1355415 tid 1 2024-03-27T09:39:33.017 INFO:tasks.rados.rados.0.smithi170.stdout:1121: writing smithi17031214-48 from 2083358 to 2801538 tid 2 2024-03-27T09:39:33.017 INFO:tasks.rados.rados.0.smithi170.stdout:1122: snap_remove snap 127 2024-03-27T09:39:33.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:33.232 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:33.230+0000 7fdec959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:33.236 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:33.235+0000 7fdec959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:33.260 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:33.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:34.005 INFO:tasks.rados.rados.0.smithi170.stdout:1122: done (1 left) 2024-03-27T09:39:34.005 INFO:tasks.rados.rados.0.smithi170.stdout:1123: read oid 12 snap 126 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1123: expect (ObjNum 539 snap 126 seq_num 539) 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1124: write oid 38 current snap is 128 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: finishing write tid 1 to smithi17031214-48 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: finishing write tid 2 to smithi17031214-48 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: finishing write tid 3 to smithi17031214-48 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: finishing write tid 4 to smithi17031214-48 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: oid 48 updating version 0 to 288 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: oid 48 updating version 288 to 289 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: oid 48 version 289 is already newer than 288 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 289 (ObjNum 542 snap 128 seq_num 542) dirty exists 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1121: left oid 48 (ObjNum 542 snap 128 seq_num 542) 2024-03-27T09:39:34.006 INFO:tasks.rados.rados.0.smithi170.stdout:1124: seq_num 543 ranges {726201=721270,2141840=578191} 2024-03-27T09:39:34.012 INFO:tasks.rados.rados.0.smithi170.stdout:1124: writing smithi17031214-38 from 726201 to 1447471 tid 1 2024-03-27T09:39:34.019 INFO:tasks.rados.rados.0.smithi170.stdout:1124: writing smithi17031214-38 from 2141840 to 2720031 tid 2 2024-03-27T09:39:34.019 INFO:tasks.rados.rados.0.smithi170.stdout:1121: done (2 left) 2024-03-27T09:39:34.019 INFO:tasks.rados.rados.0.smithi170.stdout:1125: delete oid 11 current snap is 128 2024-03-27T09:39:34.144 INFO:tasks.rados.rados.0.smithi170.stdout:1124: finishing write tid 1 to smithi17031214-38 2024-03-27T09:39:34.154 INFO:tasks.rados.rados.0.smithi170.stdout:1123: expect (ObjNum 539 snap 126 seq_num 539) 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: finishing write tid 2 to smithi17031214-38 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: finishing write tid 3 to smithi17031214-38 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: finishing write tid 4 to smithi17031214-38 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: oid 38 updating version 0 to 302 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: oid 38 version 302 is already newer than 301 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: oid 38 updating version 302 to 303 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 303 (ObjNum 543 snap 128 seq_num 543) dirty exists 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: left oid 38 (ObjNum 543 snap 128 seq_num 543) 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1123: done (2 left) 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1124: done (1 left) 2024-03-27T09:39:34.327 INFO:tasks.rados.rados.0.smithi170.stdout:1125: done (0 left) 2024-03-27T09:39:34.328 INFO:tasks.rados.rados.0.smithi170.stdout:1126: snap_remove snap 126 2024-03-27T09:39:35.007 INFO:tasks.rados.rados.0.smithi170.stdout:1126: done (0 left) 2024-03-27T09:39:35.008 INFO:tasks.rados.rados.0.smithi170.stdout:1127: write (excl) oid 7 current snap is 128 2024-03-27T09:39:35.008 INFO:tasks.rados.rados.0.smithi170.stdout:1127: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-03-27T09:39:35.014 INFO:tasks.rados.rados.0.smithi170.stdout:1127: writing smithi17031214-7 from 759815 to 1231583 tid 1 2024-03-27T09:39:35.020 INFO:tasks.rados.rados.0.smithi170.stdout:1127: writing smithi17031214-7 from 1742451 to 2207958 tid 2 2024-03-27T09:39:35.026 INFO:tasks.rados.rados.0.smithi170.stdout:1127: writing smithi17031214-7 from 2917178 to 3159815 tid 3 2024-03-27T09:39:35.026 INFO:tasks.rados.rados.0.smithi170.stdout:1128: read oid 23 snap -1 2024-03-27T09:39:35.026 INFO:tasks.rados.rados.0.smithi170.stdout:1128: expect (ObjNum 519 snap 121 seq_num 519) 2024-03-27T09:39:35.026 INFO:tasks.rados.rados.0.smithi170.stdout:1129: delete oid 15 current snap is 128 2024-03-27T09:39:35.088 INFO:tasks.rados.rados.0.smithi170.stdout:1128: expect (ObjNum 519 snap 121 seq_num 519) 2024-03-27T09:39:35.232 INFO:tasks.rados.rados.0.smithi170.stdout:1127: finishing write tid 1 to smithi17031214-7 2024-03-27T09:39:35.232 INFO:tasks.rados.rados.0.smithi170.stdout:1127: finishing write tid 2 to smithi17031214-7 2024-03-27T09:39:35.232 INFO:tasks.rados.rados.0.smithi170.stdout:1127: finishing write tid 3 to smithi17031214-7 2024-03-27T09:39:35.232 INFO:tasks.rados.rados.0.smithi170.stdout:1127: finishing write tid 4 to smithi17031214-7 2024-03-27T09:39:35.232 INFO:tasks.rados.rados.0.smithi170.stdout:1127: finishing write tid 5 to smithi17031214-7 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: oid 7 updating version 0 to 119 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: oid 7 updating version 119 to 120 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: oid 7 version 120 is already newer than 119 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: oid 7 version 120 is already newer than 117 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 120 (ObjNum 545 snap 128 seq_num 545) dirty exists 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: left oid 7 (ObjNum 545 snap 128 seq_num 545) 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1127: done (2 left) 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1128: done (1 left) 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1129: done (0 left) 2024-03-27T09:39:35.233 INFO:tasks.rados.rados.0.smithi170.stdout:1130: snap_create 2024-03-27T09:39:35.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:35.418 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:35.416+0000 7f622459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:35.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:35.421+0000 7f622459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:35.445 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:35.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:35.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:35.981 INFO:tasks.rados.rados.0.smithi170.stdout:1130: done (0 left) 2024-03-27T09:39:35.981 INFO:tasks.rados.rados.0.smithi170.stdout:1131: write oid 36 current snap is 129 2024-03-27T09:39:35.981 INFO:tasks.rados.rados.0.smithi170.stdout:1131: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-03-27T09:39:35.989 INFO:tasks.rados.rados.0.smithi170.stdout:1131: writing smithi17031214-36 from 793429 to 1415695 tid 1 2024-03-27T09:39:35.994 INFO:tasks.rados.rados.0.smithi170.stdout:1131: writing smithi17031214-36 from 1859415 to 2328591 tid 2 2024-03-27T09:39:35.999 INFO:tasks.rados.rados.0.smithi170.stdout:1131: writing smithi17031214-36 from 2972024 to 3193429 tid 3 2024-03-27T09:39:35.999 INFO:tasks.rados.rados.0.smithi170.stdout:1132: write oid 5 current snap is 129 2024-03-27T09:39:35.999 INFO:tasks.rados.rados.0.smithi170.stdout:1132: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-03-27T09:39:36.003 INFO:tasks.rados.rados.0.smithi170.stdout:1132: writing smithi17031214-5 from 410236 to 824104 tid 1 2024-03-27T09:39:36.010 INFO:tasks.rados.rados.0.smithi170.stdout:1132: writing smithi17031214-5 from 1234250 to 1963437 tid 2 2024-03-27T09:39:36.017 INFO:tasks.rados.rados.0.smithi170.stdout:1132: writing smithi17031214-5 from 2715800 to 3138863 tid 3 2024-03-27T09:39:36.017 INFO:tasks.rados.rados.0.smithi170.stdout:1133: snap_create 2024-03-27T09:39:36.035 INFO:tasks.rados.rados.0.smithi170.stdout:1132: finishing write tid 1 to smithi17031214-5 2024-03-27T09:39:36.054 INFO:tasks.rados.rados.0.smithi170.stdout:1132: finishing write tid 2 to smithi17031214-5 2024-03-27T09:39:36.072 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:36.076 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:36.086 INFO:tasks.rados.rados.0.smithi170.stdout:1131: finishing write tid 1 to smithi17031214-36 2024-03-27T09:39:36.088 INFO:tasks.rados.rados.0.smithi170.stdout:1132: finishing write tid 3 to smithi17031214-5 2024-03-27T09:39:36.094 INFO:tasks.rados.rados.0.smithi170.stdout:1132: finishing write tid 4 to smithi17031214-5 2024-03-27T09:39:36.095 INFO:tasks.rados.rados.0.smithi170.stdout:1131: finishing write tid 2 to smithi17031214-36 2024-03-27T09:39:36.097 INFO:tasks.rados.rados.0.smithi170.stdout:1132: finishing write tid 5 to smithi17031214-5 2024-03-27T09:39:36.097 INFO:tasks.rados.rados.0.smithi170.stdout:1132: oid 5 updating version 0 to 182 2024-03-27T09:39:36.097 INFO:tasks.rados.rados.0.smithi170.stdout:1132: oid 5 updating version 182 to 184 2024-03-27T09:39:36.097 INFO:tasks.rados.rados.0.smithi170.stdout:1132: oid 5 version 184 is already newer than 184 2024-03-27T09:39:36.098 INFO:tasks.rados.rados.0.smithi170.stdout:1132: oid 5 version 184 is already newer than 181 2024-03-27T09:39:36.098 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 184 (ObjNum 548 snap 129 seq_num 548) dirty exists 2024-03-27T09:39:36.098 INFO:tasks.rados.rados.0.smithi170.stdout:1132: left oid 5 (ObjNum 548 snap 129 seq_num 548) 2024-03-27T09:39:36.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~226s 2024-03-27T09:39:36.109 INFO:tasks.rados.rados.0.smithi170.stdout:1131: finishing write tid 3 to smithi17031214-36 2024-03-27T09:39:36.121 INFO:tasks.rados.rados.0.smithi170.stdout:1131: finishing write tid 4 to smithi17031214-36 2024-03-27T09:39:36.124 INFO:tasks.rados.rados.0.smithi170.stdout:1131: finishing write tid 5 to smithi17031214-36 2024-03-27T09:39:36.124 INFO:tasks.rados.rados.0.smithi170.stdout:1131: oid 36 updating version 0 to 369 2024-03-27T09:39:36.125 INFO:tasks.rados.rados.0.smithi170.stdout:1131: oid 36 version 369 is already newer than 369 2024-03-27T09:39:36.125 INFO:tasks.rados.rados.0.smithi170.stdout:1131: oid 36 updating version 369 to 371 2024-03-27T09:39:36.125 INFO:tasks.rados.rados.0.smithi170.stdout:1131: oid 36 version 371 is already newer than 370 2024-03-27T09:39:36.125 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 371 (ObjNum 547 snap 129 seq_num 547) dirty exists 2024-03-27T09:39:36.125 INFO:tasks.rados.rados.0.smithi170.stdout:1131: left oid 36 (ObjNum 547 snap 129 seq_num 547) 2024-03-27T09:39:37.014 INFO:tasks.rados.rados.0.smithi170.stdout:1131: done (2 left) 2024-03-27T09:39:37.014 INFO:tasks.rados.rados.0.smithi170.stdout:1132: done (1 left) 2024-03-27T09:39:37.014 INFO:tasks.rados.rados.0.smithi170.stdout:1133: done (0 left) 2024-03-27T09:39:37.014 INFO:tasks.rados.rados.0.smithi170.stdout:1134: write oid 34 current snap is 130 2024-03-27T09:39:37.014 INFO:tasks.rados.rados.0.smithi170.stdout:1134: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-03-27T09:39:37.018 INFO:tasks.rados.rados.0.smithi170.stdout:1134: writing smithi17031214-34 from 427043 to 916160 tid 1 2024-03-27T09:39:37.025 INFO:tasks.rados.rados.0.smithi170.stdout:1134: writing smithi17031214-34 from 1576379 to 2049224 tid 2 2024-03-27T09:39:37.032 INFO:tasks.rados.rados.0.smithi170.stdout:1134: writing smithi17031214-34 from 2626870 to 3227043 tid 3 2024-03-27T09:39:37.032 INFO:tasks.rados.rados.0.smithi170.stdout:1135: snap_create 2024-03-27T09:39:37.147 INFO:tasks.rados.rados.0.smithi170.stdout:1134: finishing write tid 1 to smithi17031214-34 2024-03-27T09:39:37.166 INFO:tasks.rados.rados.0.smithi170.stdout:1134: finishing write tid 2 to smithi17031214-34 2024-03-27T09:39:37.190 INFO:tasks.rados.rados.0.smithi170.stdout:1134: finishing write tid 3 to smithi17031214-34 2024-03-27T09:39:37.198 INFO:tasks.rados.rados.0.smithi170.stdout:1134: finishing write tid 4 to smithi17031214-34 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: finishing write tid 5 to smithi17031214-34 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: oid 34 updating version 0 to 443 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: oid 34 updating version 443 to 444 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: oid 34 updating version 444 to 445 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: oid 34 version 445 is already newer than 442 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 445 (ObjNum 549 snap 130 seq_num 549) dirty exists 2024-03-27T09:39:37.200 INFO:tasks.rados.rados.0.smithi170.stdout:1134: left oid 34 (ObjNum 549 snap 130 seq_num 549) 2024-03-27T09:39:37.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:37.602 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:37.599+0000 7fe7dfb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:37.605 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:37.605+0000 7fe7dfb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:37.629 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:37.675 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:38.017 INFO:tasks.rados.rados.0.smithi170.stdout:1134: done (1 left) 2024-03-27T09:39:38.017 INFO:tasks.rados.rados.0.smithi170.stdout:1135: done (0 left) 2024-03-27T09:39:38.017 INFO:tasks.rados.rados.0.smithi170.stdout:1136: snap_remove snap 130 2024-03-27T09:39:39.022 INFO:tasks.rados.rados.0.smithi170.stdout:1136: done (0 left) 2024-03-27T09:39:39.022 INFO:tasks.rados.rados.0.smithi170.stdout:1137: read oid 44 snap -1 2024-03-27T09:39:39.023 INFO:tasks.rados.rados.0.smithi170.stdout:1137: expect deleted 2024-03-27T09:39:39.023 INFO:tasks.rados.rados.0.smithi170.stdout:1138: write oid 12 current snap is 131 2024-03-27T09:39:39.023 INFO:tasks.rados.rados.0.smithi170.stdout:1138: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-03-27T09:39:39.028 INFO:tasks.rados.rados.0.smithi170.stdout:1138: writing smithi17031214-12 from 443850 to 1008216 tid 1 2024-03-27T09:39:39.036 INFO:tasks.rados.rados.0.smithi170.stdout:1138: writing smithi17031214-12 from 1634861 to 2367717 tid 2 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1138: writing smithi17031214-12 from 2770646 to 3243850 tid 3 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1139: read oid 25 snap -1 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1139: expect (ObjNum 499 snap 114 seq_num 499) 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1140: read oid 36 snap -1 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1140: expect (ObjNum 547 snap 129 seq_num 547) 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1141: write oid 9 current snap is 131 2024-03-27T09:39:39.043 INFO:tasks.rados.rados.0.smithi170.stdout:1141: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-03-27T09:39:39.051 INFO:tasks.rados.rados.0.smithi170.stdout:1141: writing smithi17031214-9 from 460657 to 1100272 tid 1 2024-03-27T09:39:39.060 INFO:tasks.rados.rados.0.smithi170.stdout:1141: writing smithi17031214-9 from 1693343 to 2169857 tid 2 2024-03-27T09:39:39.069 INFO:tasks.rados.rados.0.smithi170.stdout:1141: writing smithi17031214-9 from 2681716 to 3260657 tid 3 2024-03-27T09:39:39.069 INFO:tasks.rados.rados.0.smithi170.stdout:1142: read oid 22 snap 123 2024-03-27T09:39:39.069 INFO:tasks.rados.rados.0.smithi170.stdout:1142: expect (ObjNum 529 snap 122 seq_num 529) 2024-03-27T09:39:39.069 INFO:tasks.rados.rados.0.smithi170.stdout:1143: write (excl) oid 15 current snap is 131 2024-03-27T09:39:39.069 INFO:tasks.rados.rados.0.smithi170.stdout:1143: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-03-27T09:39:39.075 INFO:tasks.rados.rados.0.smithi170.stdout:1143: writing smithi17031214-15 from 477464 to 1192328 tid 1 2024-03-27T09:39:39.083 INFO:tasks.rados.rados.0.smithi170.stdout:1143: writing smithi17031214-15 from 1751825 to 2488350 tid 2 2024-03-27T09:39:39.087 INFO:tasks.rados.rados.0.smithi170.stdout:1143: writing smithi17031214-15 from 3225492 to 3277464 tid 3 2024-03-27T09:39:39.088 INFO:tasks.rados.rados.0.smithi170.stdout:1144: read oid 48 snap -1 2024-03-27T09:39:39.088 INFO:tasks.rados.rados.0.smithi170.stdout:1144: expect (ObjNum 542 snap 128 seq_num 542) 2024-03-27T09:39:39.088 INFO:tasks.rados.rados.0.smithi170.stdout:1145: write oid 35 current snap is 131 2024-03-27T09:39:39.088 INFO:tasks.rados.rados.0.smithi170.stdout:1145: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-03-27T09:39:39.098 INFO:tasks.rados.rados.0.smithi170.stdout:1145: writing smithi17031214-35 from 494271 to 1284384 tid 1 2024-03-27T09:39:39.105 INFO:tasks.rados.rados.0.smithi170.stdout:1145: writing smithi17031214-35 from 1693954 to 2174137 tid 2 2024-03-27T09:39:39.112 INFO:tasks.rados.rados.0.smithi170.stdout:1145: writing smithi17031214-35 from 2620209 to 3132338 tid 3 2024-03-27T09:39:39.113 INFO:tasks.rados.rados.0.smithi170.stdout:1146: write oid 20 current snap is 131 2024-03-27T09:39:39.113 INFO:tasks.rados.rados.0.smithi170.stdout:1146: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-03-27T09:39:39.120 INFO:tasks.rados.rados.0.smithi170.stdout:1146: writing smithi17031214-20 from 511078 to 976440 tid 1 2024-03-27T09:39:39.130 INFO:tasks.rados.rados.0.smithi170.stdout:1146: writing smithi17031214-20 from 1752436 to 2492630 tid 2 2024-03-27T09:39:39.135 INFO:tasks.rados.rados.0.smithi170.stdout:1146: writing smithi17031214-20 from 3163985 to 3311078 tid 3 2024-03-27T09:39:39.135 INFO:tasks.rados.rados.0.smithi170.stdout:1147: delete oid 13 current snap is 131 2024-03-27T09:39:39.135 INFO:tasks.rados.rados.0.smithi170.stdout:1144: expect (ObjNum 542 snap 128 seq_num 542) 2024-03-27T09:39:39.361 INFO:tasks.rados.rados.0.smithi170.stdout:1145: finishing write tid 1 to smithi17031214-35 2024-03-27T09:39:39.361 INFO:tasks.rados.rados.0.smithi170.stdout:1138: finishing write tid 1 to smithi17031214-12 2024-03-27T09:39:39.361 INFO:tasks.rados.rados.0.smithi170.stdout:1145: finishing write tid 2 to smithi17031214-35 2024-03-27T09:39:39.361 INFO:tasks.rados.rados.0.smithi170.stdout:1142: expect (ObjNum 529 snap 122 seq_num 529) 2024-03-27T09:39:39.551 INFO:tasks.rados.rados.0.smithi170.stdout:1140: expect (ObjNum 547 snap 129 seq_num 547) 2024-03-27T09:39:39.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1141: finishing write tid 1 to smithi17031214-9 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1145: finishing write tid 3 to smithi17031214-35 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1145: finishing write tid 4 to smithi17031214-35 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1137: done (10 left) 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1140: done (9 left) 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1142: done (8 left) 2024-03-27T09:39:39.716 INFO:tasks.rados.rados.0.smithi170.stdout:1144: done (7 left) 2024-03-27T09:39:39.717 INFO:tasks.rados.rados.0.smithi170.stdout:1147: done (6 left) 2024-03-27T09:39:39.717 INFO:tasks.rados.rados.0.smithi170.stdout:1148: snap_remove snap 128 2024-03-27T09:39:39.784 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:39.782+0000 7fa28435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:39.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:39.787+0000 7fa28435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:39.812 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:39.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:40.024 INFO:tasks.rados.rados.0.smithi170.stdout:1148: done (6 left) 2024-03-27T09:39:40.024 INFO:tasks.rados.rados.0.smithi170.stdout:1149: read oid 13 snap 123 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1149: expect deleted 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1150: write oid 39 current snap is 131 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: finishing write tid 5 to smithi17031214-35 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: oid 35 updating version 0 to 366 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: oid 35 updating version 366 to 369 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: oid 35 version 369 is already newer than 367 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: oid 35 version 369 is already newer than 369 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 369 (ObjNum 553 snap 131 seq_num 553) dirty exists 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1145: left oid 35 (ObjNum 553 snap 131 seq_num 553) 2024-03-27T09:39:40.025 INFO:tasks.rados.rados.0.smithi170.stdout:1150: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-03-27T09:39:40.032 INFO:tasks.rados.rados.0.smithi170.stdout:1150: writing smithi17031214-39 from 544692 to 1276905 tid 1 2024-03-27T09:39:40.042 INFO:tasks.rados.rados.0.smithi170.stdout:1150: writing smithi17031214-39 from 1985753 to 2729616 tid 2 2024-03-27T09:39:40.046 INFO:tasks.rados.rados.0.smithi170.stdout:1150: writing smithi17031214-39 from 3335184 to 3344692 tid 3 2024-03-27T09:39:40.046 INFO:tasks.rados.rados.0.smithi170.stdout:1145: done (7 left) 2024-03-27T09:39:40.046 INFO:tasks.rados.rados.0.smithi170.stdout:1151: read oid 18 snap -1 2024-03-27T09:39:40.047 INFO:tasks.rados.rados.0.smithi170.stdout:1151: expect (ObjNum 520 snap 121 seq_num 520) 2024-03-27T09:39:40.047 INFO:tasks.rados.rados.0.smithi170.stdout:1152: write (excl) oid 36 current snap is 131 2024-03-27T09:39:40.047 INFO:tasks.rados.rados.0.smithi170.stdout:1139: expect (ObjNum 499 snap 114 seq_num 499) 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1138: finishing write tid 2 to smithi17031214-12 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1143: finishing write tid 1 to smithi17031214-15 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1141: finishing write tid 2 to smithi17031214-9 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1138: finishing write tid 3 to smithi17031214-12 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1143: finishing write tid 2 to smithi17031214-15 2024-03-27T09:39:40.215 INFO:tasks.rados.rados.0.smithi170.stdout:1138: finishing write tid 4 to smithi17031214-12 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: finishing write tid 5 to smithi17031214-12 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: oid 12 updating version 0 to 340 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: oid 12 updating version 340 to 341 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: oid 12 version 341 is already newer than 339 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: oid 12 version 341 is already newer than 338 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 341 (ObjNum 550 snap 131 seq_num 550) dirty exists 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1138: left oid 12 (ObjNum 550 snap 131 seq_num 550) 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1141: finishing write tid 3 to smithi17031214-9 2024-03-27T09:39:40.216 INFO:tasks.rados.rados.0.smithi170.stdout:1152: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-03-27T09:39:40.220 INFO:tasks.rados.rados.0.smithi170.stdout:1152: writing smithi17031214-36 from 561499 to 968961 tid 1 2024-03-27T09:39:40.226 INFO:tasks.rados.rados.0.smithi170.stdout:1152: writing smithi17031214-36 from 1527882 to 2015403 tid 2 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1152: writing smithi17031214-36 from 2729901 to 3287118 tid 3 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1138: done (8 left) 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1139: done (7 left) 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1153: read oid 44 snap -1 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1153: expect deleted 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1154: write oid 27 current snap is 131 2024-03-27T09:39:40.233 INFO:tasks.rados.rados.0.smithi170.stdout:1154: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-03-27T09:39:40.239 INFO:tasks.rados.rados.0.smithi170.stdout:1154: writing smithi17031214-27 from 578306 to 1061017 tid 1 2024-03-27T09:39:40.245 INFO:tasks.rados.rados.0.smithi170.stdout:1154: writing smithi17031214-27 from 1586364 to 2217543 tid 2 2024-03-27T09:39:40.251 INFO:tasks.rados.rados.0.smithi170.stdout:1154: writing smithi17031214-27 from 2757324 to 3242166 tid 3 2024-03-27T09:39:40.251 INFO:tasks.rados.rados.0.smithi170.stdout:1155: read oid 2 snap 129 2024-03-27T09:39:40.251 INFO:tasks.rados.rados.0.smithi170.stdout:1155: expect (ObjNum 532 snap 123 seq_num 532) 2024-03-27T09:39:40.251 INFO:tasks.rados.rados.0.smithi170.stdout:1156: snap_remove snap 113 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1156: done (10 left) 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1157: read oid 26 snap 123 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1157: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1158: write oid 11 current snap is 131 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1141: finishing write tid 4 to smithi17031214-9 2024-03-27T09:39:41.027 INFO:tasks.rados.rados.0.smithi170.stdout:1158: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-03-27T09:39:41.033 INFO:tasks.rados.rados.0.smithi170.stdout:1158: writing smithi17031214-11 from 595113 to 1153073 tid 1 2024-03-27T09:39:41.039 INFO:tasks.rados.rados.0.smithi170.stdout:1158: writing smithi17031214-11 from 1644846 to 2136036 tid 2 2024-03-27T09:39:41.048 INFO:tasks.rados.rados.0.smithi170.stdout:1158: writing smithi17031214-11 from 2784747 to 3395113 tid 3 2024-03-27T09:39:41.048 INFO:tasks.rados.rados.0.smithi170.stdout:1159: read oid 47 snap -1 2024-03-27T09:39:41.048 INFO:tasks.rados.rados.0.smithi170.stdout:1159: expect (ObjNum 521 snap 121 seq_num 521) 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1160: read oid 42 snap 129 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1160: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1161: delete oid 19 current snap is 131 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: finishing write tid 5 to smithi17031214-9 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: oid 9 updating version 0 to 450 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: oid 9 version 450 is already newer than 450 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: oid 9 version 450 is already newer than 448 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: oid 9 version 450 is already newer than 447 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 450 (ObjNum 551 snap 131 seq_num 551) dirty exists 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1141: left oid 9 (ObjNum 551 snap 131 seq_num 551) 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1143: finishing write tid 3 to smithi17031214-15 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1146: finishing write tid 1 to smithi17031214-20 2024-03-27T09:39:41.049 INFO:tasks.rados.rados.0.smithi170.stdout:1146: finishing write tid 2 to smithi17031214-20 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: finishing write tid 3 to smithi17031214-20 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: finishing write tid 4 to smithi17031214-20 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: finishing write tid 5 to smithi17031214-20 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: oid 20 updating version 0 to 346 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: oid 20 version 346 is already newer than 346 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: oid 20 version 346 is already newer than 346 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: oid 20 version 346 is already newer than 346 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 346 (ObjNum 554 snap 131 seq_num 554) dirty exists 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1146: left oid 20 (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1143: finishing write tid 4 to smithi17031214-15 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1143: finishing write tid 5 to smithi17031214-15 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1143: oid 15 updating version 0 to 155 2024-03-27T09:39:41.050 INFO:tasks.rados.rados.0.smithi170.stdout:1143: oid 15 updating version 155 to 156 2024-03-27T09:39:41.051 INFO:tasks.rados.rados.0.smithi170.stdout:1143: oid 15 updating version 156 to 157 2024-03-27T09:39:41.051 INFO:tasks.rados.rados.0.smithi170.stdout:1143: oid 15 updating version 157 to 158 2024-03-27T09:39:41.051 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 158 (ObjNum 552 snap 131 seq_num 552) dirty exists 2024-03-27T09:39:41.051 INFO:tasks.rados.rados.0.smithi170.stdout:1143: left oid 15 (ObjNum 552 snap 131 seq_num 552) 2024-03-27T09:39:41.051 INFO:tasks.rados.rados.0.smithi170.stdout:1151: expect (ObjNum 520 snap 121 seq_num 520) 2024-03-27T09:39:41.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: finishing write tid 1 to smithi17031214-39 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: finishing write tid 2 to smithi17031214-39 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: finishing write tid 3 to smithi17031214-39 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: finishing write tid 4 to smithi17031214-39 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: finishing write tid 5 to smithi17031214-39 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: oid 39 updating version 0 to 186 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: oid 39 updating version 186 to 188 2024-03-27T09:39:41.181 INFO:tasks.rados.rados.0.smithi170.stdout:1150: oid 39 version 188 is already newer than 187 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1150: oid 39 updating version 188 to 189 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 189 (ObjNum 556 snap 131 seq_num 556) dirty exists 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1150: left oid 39 (ObjNum 556 snap 131 seq_num 556) 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1152: finishing write tid 1 to smithi17031214-36 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: finishing write tid 1 to smithi17031214-27 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: finishing write tid 2 to smithi17031214-27 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1152: finishing write tid 2 to smithi17031214-36 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: finishing write tid 3 to smithi17031214-27 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1152: finishing write tid 3 to smithi17031214-36 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: finishing write tid 4 to smithi17031214-27 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: finishing write tid 5 to smithi17031214-27 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: oid 27 updating version 0 to 372 2024-03-27T09:39:41.182 INFO:tasks.rados.rados.0.smithi170.stdout:1154: oid 27 updating version 372 to 373 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1154: oid 27 version 373 is already newer than 372 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1154: oid 27 updating version 373 to 374 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 374 (ObjNum 558 snap 131 seq_num 558) dirty exists 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1154: left oid 27 (ObjNum 558 snap 131 seq_num 558) 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: finishing write tid 4 to smithi17031214-36 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: finishing write tid 5 to smithi17031214-36 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: oid 36 updating version 0 to 377 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: oid 36 version 377 is already newer than 377 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: oid 36 version 377 is already newer than 375 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: oid 36 updating version 377 to 378 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 378 (ObjNum 557 snap 131 seq_num 557) dirty exists 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1152: left oid 36 (ObjNum 557 snap 131 seq_num 557) 2024-03-27T09:39:41.183 INFO:tasks.rados.rados.0.smithi170.stdout:1155: expect (ObjNum 532 snap 123 seq_num 532) 2024-03-27T09:39:41.213 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:41.217 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:41.301 INFO:tasks.rados.rados.0.smithi170.stdout:1160: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:41.418 INFO:tasks.rados.rados.0.smithi170.stdout:1159: expect (ObjNum 521 snap 121 seq_num 521) 2024-03-27T09:39:41.520 INFO:tasks.rados.rados.0.smithi170.stdout:1158: finishing write tid 1 to smithi17031214-11 2024-03-27T09:39:41.520 INFO:tasks.rados.rados.0.smithi170.stdout:1157: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:39:41.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1158: finishing write tid 2 to smithi17031214-11 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1141: done (14 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1143: done (13 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1146: done (12 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1149: done (11 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1150: done (10 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1151: done (9 left) 2024-03-27T09:39:41.740 INFO:tasks.rados.rados.0.smithi170.stdout:1152: done (8 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1153: done (7 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1154: done (6 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1155: done (5 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1157: done (4 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1159: done (3 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1160: done (2 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1161: done (1 left) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1162: read oid 35 snap -1 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1162: expect (ObjNum 553 snap 131 seq_num 553) 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1163: write oid 31 current snap is 131 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1158: finishing write tid 3 to smithi17031214-11 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1158: finishing write tid 4 to smithi17031214-11 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1158: finishing write tid 5 to smithi17031214-11 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1158: oid 11 updating version 0 to 358 2024-03-27T09:39:41.741 INFO:tasks.rados.rados.0.smithi170.stdout:1158: oid 11 version 358 is already newer than 358 2024-03-27T09:39:41.742 INFO:tasks.rados.rados.0.smithi170.stdout:1158: oid 11 version 358 is already newer than 355 2024-03-27T09:39:41.742 INFO:tasks.rados.rados.0.smithi170.stdout:1158: oid 11 version 358 is already newer than 356 2024-03-27T09:39:41.742 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 358 (ObjNum 559 snap 131 seq_num 559) dirty exists 2024-03-27T09:39:41.742 INFO:tasks.rados.rados.0.smithi170.stdout:1158: left oid 11 (ObjNum 559 snap 131 seq_num 559) 2024-03-27T09:39:41.742 INFO:tasks.rados.rados.0.smithi170.stdout:1163: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-03-27T09:39:41.749 INFO:tasks.rados.rados.0.smithi170.stdout:1163: writing smithi17031214-31 from 628727 to 1337185 tid 1 2024-03-27T09:39:41.758 INFO:tasks.rados.rados.0.smithi170.stdout:1163: writing smithi17031214-31 from 2045457 to 2540316 tid 2 2024-03-27T09:39:41.764 INFO:tasks.rados.rados.0.smithi170.stdout:1163: writing smithi17031214-31 from 3123240 to 3428727 tid 3 2024-03-27T09:39:41.764 INFO:tasks.rados.rados.0.smithi170.stdout:1158: done (2 left) 2024-03-27T09:39:41.764 INFO:tasks.rados.rados.0.smithi170.stdout:1164: write (excl) oid 45 current snap is 131 2024-03-27T09:39:41.764 INFO:tasks.rados.rados.0.smithi170.stdout:1164: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-03-27T09:39:41.772 INFO:tasks.rados.rados.0.smithi170.stdout:1164: writing smithi17031214-45 from 645534 to 1429241 tid 1 2024-03-27T09:39:41.779 INFO:tasks.rados.rados.0.smithi170.stdout:1164: writing smithi17031214-45 from 2103939 to 2742456 tid 2 2024-03-27T09:39:41.783 INFO:tasks.rados.rados.0.smithi170.stdout:1164: writing smithi17031214-45 from 3150663 to 3445534 tid 3 2024-03-27T09:39:41.783 INFO:tasks.rados.rados.0.smithi170.stdout:1165: snap_remove snap 123 2024-03-27T09:39:41.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:41.970 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:41.968+0000 7f0c89c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:41.973+0000 7f0c89c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:41.998 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:42.030 INFO:tasks.rados.rados.0.smithi170.stdout:1165: done (3 left) 2024-03-27T09:39:42.030 INFO:tasks.rados.rados.0.smithi170.stdout:1166: delete oid 34 current snap is 131 2024-03-27T09:39:42.030 INFO:tasks.rados.rados.0.smithi170.stdout:1162: expect (ObjNum 553 snap 131 seq_num 553) 2024-03-27T09:39:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:42.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: finishing write tid 1 to smithi17031214-31 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: finishing write tid 2 to smithi17031214-31 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1164: finishing write tid 1 to smithi17031214-45 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: finishing write tid 3 to smithi17031214-31 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: finishing write tid 4 to smithi17031214-31 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: finishing write tid 5 to smithi17031214-31 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: oid 31 updating version 0 to 364 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: oid 31 version 364 is already newer than 361 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: oid 31 version 364 is already newer than 364 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:1163: oid 31 version 364 is already newer than 362 2024-03-27T09:39:42.199 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 364 (ObjNum 561 snap 131 seq_num 561) dirty exists 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1163: left oid 31 (ObjNum 561 snap 131 seq_num 561) 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: finishing write tid 2 to smithi17031214-45 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: finishing write tid 3 to smithi17031214-45 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1162: done (3 left) 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1163: done (2 left) 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1166: done (1 left) 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1167: snap_create 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: finishing write tid 4 to smithi17031214-45 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: finishing write tid 5 to smithi17031214-45 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: oid 45 updating version 0 to 384 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: oid 45 updating version 384 to 385 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: oid 45 version 385 is already newer than 382 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:1164: oid 45 version 385 is already newer than 384 2024-03-27T09:39:42.200 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 385 (ObjNum 562 snap 131 seq_num 562) dirty exists 2024-03-27T09:39:42.201 INFO:tasks.rados.rados.0.smithi170.stdout:1164: left oid 45 (ObjNum 562 snap 131 seq_num 562) 2024-03-27T09:39:43.033 INFO:tasks.rados.rados.0.smithi170.stdout:1164: done (1 left) 2024-03-27T09:39:43.034 INFO:tasks.rados.rados.0.smithi170.stdout:1167: done (0 left) 2024-03-27T09:39:43.034 INFO:tasks.rados.rados.0.smithi170.stdout:1168: write (excl) oid 9 current snap is 132 2024-03-27T09:39:43.034 INFO:tasks.rados.rados.0.smithi170.stdout:1168: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-03-27T09:39:43.040 INFO:tasks.rados.rados.0.smithi170.stdout:1168: writing smithi17031214-9 from 679148 to 1329706 tid 1 2024-03-27T09:39:43.048 INFO:tasks.rados.rados.0.smithi170.stdout:1168: writing smithi17031214-9 from 1937256 to 2579442 tid 2 2024-03-27T09:39:43.053 INFO:tasks.rados.rados.0.smithi170.stdout:1168: writing smithi17031214-9 from 3321862 to 3479148 tid 3 2024-03-27T09:39:43.053 INFO:tasks.rados.rados.0.smithi170.stdout:1169: delete oid 39 current snap is 132 2024-03-27T09:39:43.175 INFO:tasks.rados.rados.0.smithi170.stdout:1168: finishing write tid 1 to smithi17031214-9 2024-03-27T09:39:43.176 INFO:tasks.rados.rados.0.smithi170.stdout:1169: done (1 left) 2024-03-27T09:39:43.176 INFO:tasks.rados.rados.0.smithi170.stdout:1170: read oid 19 snap -1 2024-03-27T09:39:43.176 INFO:tasks.rados.rados.0.smithi170.stdout:1170: expect deleted 2024-03-27T09:39:43.176 INFO:tasks.rados.rados.0.smithi170.stdout:1171: snap_remove snap 129 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1171: done (2 left) 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1172: snap_create 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1168: finishing write tid 2 to smithi17031214-9 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1168: finishing write tid 3 to smithi17031214-9 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1168: finishing write tid 4 to smithi17031214-9 2024-03-27T09:39:44.038 INFO:tasks.rados.rados.0.smithi170.stdout:1168: finishing write tid 5 to smithi17031214-9 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:1168: oid 9 updating version 0 to 465 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:1168: oid 9 updating version 465 to 468 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:1168: oid 9 version 468 is already newer than 466 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:1168: oid 9 version 468 is already newer than 468 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 468 (ObjNum 564 snap 132 seq_num 564) dirty exists 2024-03-27T09:39:44.039 INFO:tasks.rados.rados.0.smithi170.stdout:1168: left oid 9 (ObjNum 564 snap 132 seq_num 564) 2024-03-27T09:39:44.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:44.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:44.159+0000 7ff3584ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:44.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:44.165+0000 7ff3584ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:44.190 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:44.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:45.038 INFO:tasks.rados.rados.0.smithi170.stdout:1168: done (2 left) 2024-03-27T09:39:45.039 INFO:tasks.rados.rados.0.smithi170.stdout:1170: done (1 left) 2024-03-27T09:39:45.039 INFO:tasks.rados.rados.0.smithi170.stdout:1172: done (0 left) 2024-03-27T09:39:45.039 INFO:tasks.rados.rados.0.smithi170.stdout:1173: snap_create 2024-03-27T09:39:45.984 INFO:tasks.rados.rados.0.smithi170.stdout:1173: done (0 left) 2024-03-27T09:39:45.984 INFO:tasks.rados.rados.0.smithi170.stdout:1174: write oid 23 current snap is 134 2024-03-27T09:39:45.984 INFO:tasks.rados.rados.0.smithi170.stdout:1174: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-03-27T09:39:45.990 INFO:tasks.rados.rados.0.smithi170.stdout:1174: writing smithi17031214-23 from 712762 to 1113818 tid 1 2024-03-27T09:39:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1174: writing smithi17031214-23 from 1537867 to 2183722 tid 2 2024-03-27T09:39:46.004 INFO:tasks.rados.rados.0.smithi170.stdout:1174: writing smithi17031214-23 from 2860355 to 3512762 tid 3 2024-03-27T09:39:46.005 INFO:tasks.rados.rados.0.smithi170.stdout:1175: snap_create 2024-03-27T09:39:46.071 INFO:tasks.rados.rados.0.smithi170.stdout:1174: finishing write tid 1 to smithi17031214-23 2024-03-27T09:39:46.096 INFO:tasks.rados.rados.0.smithi170.stdout:1174: finishing write tid 2 to smithi17031214-23 2024-03-27T09:39:46.117 INFO:tasks.rados.rados.0.smithi170.stdout:1174: finishing write tid 3 to smithi17031214-23 2024-03-27T09:39:46.127 INFO:tasks.rados.rados.0.smithi170.stdout:1174: finishing write tid 4 to smithi17031214-23 2024-03-27T09:39:46.128 INFO:tasks.rados.rados.0.smithi170.stdout:1174: finishing write tid 5 to smithi17031214-23 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:1174: oid 23 updating version 0 to 476 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:1174: oid 23 version 476 is already newer than 474 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:1174: oid 23 updating version 476 to 477 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:1174: oid 23 version 477 is already newer than 475 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 477 (ObjNum 566 snap 134 seq_num 566) dirty exists 2024-03-27T09:39:46.129 INFO:tasks.rados.rados.0.smithi170.stdout:1174: left oid 23 (ObjNum 566 snap 134 seq_num 566) 2024-03-27T09:39:46.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:46.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:46.319 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:46.322 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:46.371 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:46.369+0000 7f0936c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:46.375 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:46.374+0000 7f0936c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:46.400 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:46.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:46.984 INFO:tasks.rados.rados.0.smithi170.stdout:1174: done (1 left) 2024-03-27T09:39:46.985 INFO:tasks.rados.rados.0.smithi170.stdout:1175: done (0 left) 2024-03-27T09:39:46.985 INFO:tasks.rados.rados.0.smithi170.stdout:1176: snap_remove snap 132 2024-03-27T09:39:47.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~237s 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1176: done (0 left) 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1177: read oid 15 snap 131 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1177: expect (ObjNum 552 snap 131 seq_num 552) 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1178: read oid 4 snap 131 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1178: expect (ObjNum 505 snap 116 seq_num 505) 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1179: write oid 39 current snap is 135 2024-03-27T09:39:47.986 INFO:tasks.rados.rados.0.smithi170.stdout:1179: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-03-27T09:39:47.992 INFO:tasks.rados.rados.0.smithi170.stdout:1179: writing smithi17031214-39 from 729569 to 1205874 tid 1 2024-03-27T09:39:47.999 INFO:tasks.rados.rados.0.smithi170.stdout:1179: writing smithi17031214-39 from 1996349 to 2502215 tid 2 2024-03-27T09:39:48.006 INFO:tasks.rados.rados.0.smithi170.stdout:1179: writing smithi17031214-39 from 3004131 to 3529569 tid 3 2024-03-27T09:39:48.006 INFO:tasks.rados.rados.0.smithi170.stdout:1180: snap_create 2024-03-27T09:39:48.064 INFO:tasks.rados.rados.0.smithi170.stdout:1177: expect (ObjNum 552 snap 131 seq_num 552) 2024-03-27T09:39:48.194 INFO:tasks.rados.rados.0.smithi170.stdout:1179: finishing write tid 1 to smithi17031214-39 2024-03-27T09:39:48.194 INFO:tasks.rados.rados.0.smithi170.stdout:1178: expect (ObjNum 505 snap 116 seq_num 505) 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: finishing write tid 2 to smithi17031214-39 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: finishing write tid 3 to smithi17031214-39 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: finishing write tid 4 to smithi17031214-39 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: finishing write tid 5 to smithi17031214-39 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: oid 39 updating version 0 to 200 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: oid 39 updating version 200 to 201 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: oid 39 updating version 201 to 203 2024-03-27T09:39:48.346 INFO:tasks.rados.rados.0.smithi170.stdout:1179: oid 39 version 203 is already newer than 203 2024-03-27T09:39:48.347 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 203 (ObjNum 567 snap 135 seq_num 567) dirty exists 2024-03-27T09:39:48.347 INFO:tasks.rados.rados.0.smithi170.stdout:1179: left oid 39 (ObjNum 567 snap 135 seq_num 567) 2024-03-27T09:39:48.448 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:48.557 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:48.554+0000 7fdedd006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:48.560 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:48.559+0000 7fdedd006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:48.584 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:48.632 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:48.988 INFO:tasks.rados.rados.0.smithi170.stdout:1177: done (3 left) 2024-03-27T09:39:48.988 INFO:tasks.rados.rados.0.smithi170.stdout:1178: done (2 left) 2024-03-27T09:39:48.989 INFO:tasks.rados.rados.0.smithi170.stdout:1179: done (1 left) 2024-03-27T09:39:48.989 INFO:tasks.rados.rados.0.smithi170.stdout:1180: done (0 left) 2024-03-27T09:39:48.989 INFO:tasks.rados.rados.0.smithi170.stdout:1181: snap_remove snap 133 2024-03-27T09:39:49.993 INFO:tasks.rados.rados.0.smithi170.stdout:1181: done (0 left) 2024-03-27T09:39:49.993 INFO:tasks.rados.rados.0.smithi170.stdout:1182: write (excl) oid 32 current snap is 136 2024-03-27T09:39:49.993 INFO:tasks.rados.rados.0.smithi170.stdout:1182: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-03-27T09:39:49.999 INFO:tasks.rados.rados.0.smithi170.stdout:1182: writing smithi17031214-32 from 746376 to 1297930 tid 1 2024-03-27T09:39:50.007 INFO:tasks.rados.rados.0.smithi170.stdout:1182: writing smithi17031214-32 from 2054831 to 2704355 tid 2 2024-03-27T09:39:50.012 INFO:tasks.rados.rados.0.smithi170.stdout:1182: writing smithi17031214-32 from 3315201 to 3546376 tid 3 2024-03-27T09:39:50.012 INFO:tasks.rados.rados.0.smithi170.stdout:1183: snap_remove snap 131 2024-03-27T09:39:50.633 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:50.741 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:50.739+0000 7ff1c04ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:50.745 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:50.744+0000 7ff1c04ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:50.769 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:50.819 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:50.995 INFO:tasks.rados.rados.0.smithi170.stdout:1183: done (1 left) 2024-03-27T09:39:50.995 INFO:tasks.rados.rados.0.smithi170.stdout:1184: snap_remove snap 135 2024-03-27T09:39:51.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:51.422 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:51.426 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1184: done (1 left) 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1185: delete oid 37 current snap is 136 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1182: finishing write tid 1 to smithi17031214-32 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1182: finishing write tid 2 to smithi17031214-32 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1182: finishing write tid 3 to smithi17031214-32 2024-03-27T09:39:51.999 INFO:tasks.rados.rados.0.smithi170.stdout:1182: finishing write tid 4 to smithi17031214-32 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: finishing write tid 5 to smithi17031214-32 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: oid 32 updating version 0 to 375 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: oid 32 updating version 375 to 376 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: oid 32 version 376 is already newer than 375 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: oid 32 version 376 is already newer than 373 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 376 (ObjNum 568 snap 136 seq_num 568) dirty exists 2024-03-27T09:39:52.000 INFO:tasks.rados.rados.0.smithi170.stdout:1182: left oid 32 (ObjNum 568 snap 136 seq_num 568) 2024-03-27T09:39:52.100 INFO:tasks.rados.rados.0.smithi170.stdout:1182: done (1 left) 2024-03-27T09:39:52.100 INFO:tasks.rados.rados.0.smithi170.stdout:1185: done (0 left) 2024-03-27T09:39:52.100 INFO:tasks.rados.rados.0.smithi170.stdout:1186: write oid 25 current snap is 136 2024-03-27T09:39:52.100 INFO:tasks.rados.rados.0.smithi170.stdout:1186: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-03-27T09:39:52.107 INFO:tasks.rados.rados.0.smithi170.stdout:1186: writing smithi17031214-25 from 779990 to 1482042 tid 1 2024-03-27T09:39:52.114 INFO:tasks.rados.rados.0.smithi170.stdout:1186: writing smithi17031214-25 from 2055442 to 2708635 tid 2 2024-03-27T09:39:52.119 INFO:tasks.rados.rados.0.smithi170.stdout:1186: writing smithi17031214-25 from 3253694 to 3579990 tid 3 2024-03-27T09:39:52.119 INFO:tasks.rados.rados.0.smithi170.stdout:1187: read oid 48 snap -1 2024-03-27T09:39:52.119 INFO:tasks.rados.rados.0.smithi170.stdout:1187: expect (ObjNum 542 snap 128 seq_num 542) 2024-03-27T09:39:52.119 INFO:tasks.rados.rados.0.smithi170.stdout:1188: delete oid 7 current snap is 136 2024-03-27T09:39:52.147 INFO:tasks.rados.rados.0.smithi170.stdout:1187: expect (ObjNum 542 snap 128 seq_num 542) 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: finishing write tid 1 to smithi17031214-25 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: finishing write tid 2 to smithi17031214-25 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: finishing write tid 3 to smithi17031214-25 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: finishing write tid 4 to smithi17031214-25 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: finishing write tid 5 to smithi17031214-25 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: oid 25 updating version 0 to 400 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: oid 25 version 400 is already newer than 398 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: oid 25 version 400 is already newer than 397 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:1186: oid 25 version 400 is already newer than 399 2024-03-27T09:39:52.350 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 400 (ObjNum 570 snap 136 seq_num 570) dirty exists 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1186: left oid 25 (ObjNum 570 snap 136 seq_num 570) 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1186: done (2 left) 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1187: done (1 left) 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1188: done (0 left) 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1189: write (excl) oid 6 current snap is 136 2024-03-27T09:39:52.351 INFO:tasks.rados.rados.0.smithi170.stdout:1189: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-03-27T09:39:52.355 INFO:tasks.rados.rados.0.smithi170.stdout:1189: writing smithi17031214-6 from 413604 to 982507 tid 1 2024-03-27T09:39:52.361 INFO:tasks.rados.rados.0.smithi170.stdout:1189: writing smithi17031214-6 from 1488759 to 2145621 tid 2 2024-03-27T09:39:52.366 INFO:tasks.rados.rados.0.smithi170.stdout:1189: writing smithi17031214-6 from 2624893 to 3216602 tid 3 2024-03-27T09:39:52.367 INFO:tasks.rados.rados.0.smithi170.stdout:1190: snap_create 2024-03-27T09:39:52.389 INFO:tasks.rados.rados.0.smithi170.stdout:1189: finishing write tid 1 to smithi17031214-6 2024-03-27T09:39:52.412 INFO:tasks.rados.rados.0.smithi170.stdout:1189: finishing write tid 2 to smithi17031214-6 2024-03-27T09:39:52.425 INFO:tasks.rados.rados.0.smithi170.stdout:1189: finishing write tid 3 to smithi17031214-6 2024-03-27T09:39:52.433 INFO:tasks.rados.rados.0.smithi170.stdout:1189: finishing write tid 4 to smithi17031214-6 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:1189: finishing write tid 5 to smithi17031214-6 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:1189: oid 6 updating version 0 to 211 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:1189: oid 6 version 211 is already newer than 209 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:1189: oid 6 version 211 is already newer than 211 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:1189: oid 6 updating version 211 to 212 2024-03-27T09:39:52.435 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 212 (ObjNum 572 snap 136 seq_num 572) dirty exists 2024-03-27T09:39:52.436 INFO:tasks.rados.rados.0.smithi170.stdout:1189: left oid 6 (ObjNum 572 snap 136 seq_num 572) 2024-03-27T09:39:52.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~242s 2024-03-27T09:39:52.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:52.928 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:52.926+0000 7f3c75fbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:52.932 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:52.931+0000 7f3c75fbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:52.957 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1189: done (1 left) 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1190: done (0 left) 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1191: read oid 31 snap 134 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1191: expect (ObjNum 561 snap 131 seq_num 561) 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1192: write (excl) oid 45 current snap is 137 2024-03-27T09:39:53.002 INFO:tasks.rados.rados.0.smithi170.stdout:1192: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-03-27T09:39:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:53.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:53.008 INFO:tasks.rados.rados.0.smithi170.stdout:1192: writing smithi17031214-45 from 430411 to 1074563 tid 1 2024-03-27T09:39:53.014 INFO:tasks.rados.rados.0.smithi170.stdout:1192: writing smithi17031214-45 from 1830888 to 2231408 tid 2 2024-03-27T09:39:53.023 INFO:tasks.rados.rados.0.smithi170.stdout:1192: writing smithi17031214-45 from 2935963 to 3338944 tid 3 2024-03-27T09:39:53.023 INFO:tasks.rados.rados.0.smithi170.stdout:1193: write (excl) oid 17 current snap is 137 2024-03-27T09:39:53.023 INFO:tasks.rados.rados.0.smithi170.stdout:1193: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-03-27T09:39:53.032 INFO:tasks.rados.rados.0.smithi170.stdout:1193: writing smithi17031214-17 from 447218 to 1166619 tid 1 2024-03-27T09:39:53.043 INFO:tasks.rados.rados.0.smithi170.stdout:1193: writing smithi17031214-17 from 1889370 to 2549901 tid 2 2024-03-27T09:39:53.048 INFO:tasks.rados.rados.0.smithi170.stdout:1193: writing smithi17031214-17 from 2963386 to 3577639 tid 3 2024-03-27T09:39:53.049 INFO:tasks.rados.rados.0.smithi170.stdout:1194: write oid 2 current snap is 137 2024-03-27T09:39:53.049 INFO:tasks.rados.rados.0.smithi170.stdout:1194: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-03-27T09:39:53.055 INFO:tasks.rados.rados.0.smithi170.stdout:1194: writing smithi17031214-2 from 464025 to 1258675 tid 1 2024-03-27T09:39:53.061 INFO:tasks.rados.rados.0.smithi170.stdout:1194: writing smithi17031214-2 from 1947852 to 2352041 tid 2 2024-03-27T09:39:53.067 INFO:tasks.rados.rados.0.smithi170.stdout:1194: writing smithi17031214-2 from 2990809 to 3416334 tid 3 2024-03-27T09:39:53.067 INFO:tasks.rados.rados.0.smithi170.stdout:1195: write (excl) oid 33 current snap is 137 2024-03-27T09:39:53.067 INFO:tasks.rados.rados.0.smithi170.stdout:1195: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-03-27T09:39:53.072 INFO:tasks.rados.rados.0.smithi170.stdout:1195: writing smithi17031214-33 from 480832 to 950731 tid 1 2024-03-27T09:39:53.082 INFO:tasks.rados.rados.0.smithi170.stdout:1195: writing smithi17031214-33 from 1606334 to 2270534 tid 2 2024-03-27T09:39:53.091 INFO:tasks.rados.rados.0.smithi170.stdout:1195: writing smithi17031214-33 from 3018232 to 3680832 tid 3 2024-03-27T09:39:53.092 INFO:tasks.rados.rados.0.smithi170.stdout:1196: snap_remove snap 136 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1196: done (5 left) 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1197: read oid 42 snap -1 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1197: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1198: read oid 22 snap -1 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1198: expect (ObjNum 529 snap 122 seq_num 529) 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1199: delete oid 44 current snap is 137 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1193: finishing write tid 1 to smithi17031214-17 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1192: finishing write tid 1 to smithi17031214-45 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1195: finishing write tid 1 to smithi17031214-33 2024-03-27T09:39:54.007 INFO:tasks.rados.rados.0.smithi170.stdout:1191: expect (ObjNum 561 snap 131 seq_num 561) 2024-03-27T09:39:54.144 INFO:tasks.rados.rados.0.smithi170.stdout:1192: finishing write tid 2 to smithi17031214-45 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1195: finishing write tid 2 to smithi17031214-33 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1193: finishing write tid 2 to smithi17031214-17 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1191: done (7 left) 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1199: done (6 left) 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1200: read oid 20 snap -1 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1200: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1201: snap_create 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1192: finishing write tid 3 to smithi17031214-45 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1193: finishing write tid 3 to smithi17031214-17 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1195: finishing write tid 3 to smithi17031214-33 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1192: finishing write tid 4 to smithi17031214-45 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1192: finishing write tid 5 to smithi17031214-45 2024-03-27T09:39:54.145 INFO:tasks.rados.rados.0.smithi170.stdout:1192: oid 45 updating version 0 to 404 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1192: oid 45 version 404 is already newer than 402 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1192: oid 45 version 404 is already newer than 403 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1192: oid 45 updating version 404 to 405 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 405 (ObjNum 573 snap 137 seq_num 573) dirty exists 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1192: left oid 45 (ObjNum 573 snap 137 seq_num 573) 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: finishing write tid 4 to smithi17031214-33 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: finishing write tid 5 to smithi17031214-33 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: oid 33 updating version 0 to 299 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: oid 33 updating version 299 to 302 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: oid 33 version 302 is already newer than 300 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: oid 33 version 302 is already newer than 301 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 302 (ObjNum 576 snap 137 seq_num 576) dirty exists 2024-03-27T09:39:54.146 INFO:tasks.rados.rados.0.smithi170.stdout:1195: left oid 33 (ObjNum 576 snap 137 seq_num 576) 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: finishing write tid 4 to smithi17031214-17 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: finishing write tid 5 to smithi17031214-17 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: oid 17 updating version 0 to 388 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: oid 17 version 388 is already newer than 387 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: oid 17 version 388 is already newer than 385 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: oid 17 version 388 is already newer than 386 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 388 (ObjNum 574 snap 137 seq_num 574) dirty exists 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1193: left oid 17 (ObjNum 574 snap 137 seq_num 574) 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1194: finishing write tid 1 to smithi17031214-2 2024-03-27T09:39:54.147 INFO:tasks.rados.rados.0.smithi170.stdout:1194: finishing write tid 2 to smithi17031214-2 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: finishing write tid 3 to smithi17031214-2 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: finishing write tid 4 to smithi17031214-2 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: finishing write tid 5 to smithi17031214-2 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: oid 2 updating version 0 to 410 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: oid 2 version 410 is already newer than 410 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: oid 2 version 410 is already newer than 410 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: oid 2 version 410 is already newer than 410 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 410 (ObjNum 575 snap 137 seq_num 575) dirty exists 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1194: left oid 2 (ObjNum 575 snap 137 seq_num 575) 2024-03-27T09:39:54.148 INFO:tasks.rados.rados.0.smithi170.stdout:1198: expect (ObjNum 529 snap 122 seq_num 529) 2024-03-27T09:39:54.334 INFO:tasks.rados.rados.0.smithi170.stdout:1197: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:54.450 INFO:tasks.rados.rados.0.smithi170.stdout:1200: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1192: done (7 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1193: done (6 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1194: done (5 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1195: done (4 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1197: done (3 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1198: done (2 left) 2024-03-27T09:39:55.007 INFO:tasks.rados.rados.0.smithi170.stdout:1200: done (1 left) 2024-03-27T09:39:55.008 INFO:tasks.rados.rados.0.smithi170.stdout:1201: done (0 left) 2024-03-27T09:39:55.008 INFO:tasks.rados.rados.0.smithi170.stdout:1202: write (excl) oid 7 current snap is 138 2024-03-27T09:39:55.008 INFO:tasks.rados.rados.0.smithi170.stdout:1202: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-03-27T09:39:55.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:55.013 INFO:tasks.rados.rados.0.smithi170.stdout:1202: writing smithi17031214-7 from 514446 to 1251196 tid 1 2024-03-27T09:39:55.020 INFO:tasks.rados.rados.0.smithi170.stdout:1202: writing smithi17031214-7 from 1723298 to 2391167 tid 2 2024-03-27T09:39:55.027 INFO:tasks.rados.rados.0.smithi170.stdout:1202: writing smithi17031214-7 from 3073078 to 3714446 tid 3 2024-03-27T09:39:55.027 INFO:tasks.rados.rados.0.smithi170.stdout:1203: delete oid 36 current snap is 138 2024-03-27T09:39:55.085 INFO:tasks.rados.rados.0.smithi170.stdout:1202: finishing write tid 1 to smithi17031214-7 2024-03-27T09:39:55.087 INFO:tasks.rados.rados.0.smithi170.stdout:1203: done (1 left) 2024-03-27T09:39:55.087 INFO:tasks.rados.rados.0.smithi170.stdout:1204: write (excl) oid 8 current snap is 138 2024-03-27T09:39:55.087 INFO:tasks.rados.rados.0.smithi170.stdout:1204: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-03-27T09:39:55.092 INFO:tasks.rados.rados.0.smithi170.stdout:1204: writing smithi17031214-8 from 548060 to 1035308 tid 1 2024-03-27T09:39:55.099 INFO:tasks.rados.rados.0.smithi170.stdout:1204: writing smithi17031214-8 from 1440262 to 2111800 tid 2 2024-03-27T09:39:55.107 INFO:tasks.rados.rados.0.smithi170.stdout:1204: writing smithi17031214-8 from 2844277 to 3642515 tid 3 2024-03-27T09:39:55.107 INFO:tasks.rados.rados.0.smithi170.stdout:1205: delete oid 14 current snap is 138 2024-03-27T09:39:55.107 INFO:tasks.rados.rados.0.smithi170.stdout:1202: finishing write tid 2 to smithi17031214-7 2024-03-27T09:39:55.120 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:55.117+0000 7fb7e995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:55.125 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:55.124+0000 7fb7e995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:55.126 INFO:tasks.rados.rados.0.smithi170.stdout:1202: finishing write tid 3 to smithi17031214-7 2024-03-27T09:39:55.135 INFO:tasks.rados.rados.0.smithi170.stdout:1202: finishing write tid 4 to smithi17031214-7 2024-03-27T09:39:55.136 INFO:tasks.rados.rados.0.smithi170.stdout:1202: finishing write tid 5 to smithi17031214-7 2024-03-27T09:39:55.136 INFO:tasks.rados.rados.0.smithi170.stdout:1202: oid 7 updating version 0 to 125 2024-03-27T09:39:55.137 INFO:tasks.rados.rados.0.smithi170.stdout:1202: oid 7 version 125 is already newer than 123 2024-03-27T09:39:55.137 INFO:tasks.rados.rados.0.smithi170.stdout:1202: oid 7 version 125 is already newer than 124 2024-03-27T09:39:55.137 INFO:tasks.rados.rados.0.smithi170.stdout:1202: oid 7 updating version 125 to 126 2024-03-27T09:39:55.137 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 126 (ObjNum 578 snap 138 seq_num 578) dirty exists 2024-03-27T09:39:55.137 INFO:tasks.rados.rados.0.smithi170.stdout:1202: left oid 7 (ObjNum 578 snap 138 seq_num 578) 2024-03-27T09:39:55.138 INFO:tasks.rados.rados.0.smithi170.stdout:1204: finishing write tid 1 to smithi17031214-8 2024-03-27T09:39:55.168 INFO:tasks.rados.rados.0.smithi170.stdout:1204: finishing write tid 2 to smithi17031214-8 2024-03-27T09:39:55.189 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:55.207 INFO:tasks.rados.rados.0.smithi170.stdout:1204: finishing write tid 3 to smithi17031214-8 2024-03-27T09:39:55.209 INFO:tasks.rados.rados.0.smithi170.stdout:1204: finishing write tid 4 to smithi17031214-8 2024-03-27T09:39:55.211 INFO:tasks.rados.rados.0.smithi170.stdout:1202: done (2 left) 2024-03-27T09:39:55.211 INFO:tasks.rados.rados.0.smithi170.stdout:1205: done (1 left) 2024-03-27T09:39:55.211 INFO:tasks.rados.rados.0.smithi170.stdout:1206: write oid 40 current snap is 138 2024-03-27T09:39:55.212 INFO:tasks.rados.rados.0.smithi170.stdout:1206: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-03-27T09:39:55.218 INFO:tasks.rados.rados.0.smithi170.stdout:1206: writing smithi17031214-40 from 581674 to 1219420 tid 1 2024-03-27T09:39:55.225 INFO:tasks.rados.rados.0.smithi170.stdout:1206: writing smithi17031214-40 from 1840873 to 2399727 tid 2 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1206: writing smithi17031214-40 from 3066417 to 3603552 tid 3 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1207: read oid 42 snap -1 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1207: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1208: read oid 20 snap -1 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1208: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1209: write oid 10 current snap is 138 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1204: finishing write tid 5 to smithi17031214-8 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1204: oid 8 updating version 0 to 396 2024-03-27T09:39:55.235 INFO:tasks.rados.rados.0.smithi170.stdout:1204: oid 8 version 396 is already newer than 393 2024-03-27T09:39:55.236 INFO:tasks.rados.rados.0.smithi170.stdout:1204: oid 8 version 396 is already newer than 396 2024-03-27T09:39:55.236 INFO:tasks.rados.rados.0.smithi170.stdout:1204: oid 8 version 396 is already newer than 394 2024-03-27T09:39:55.236 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 396 (ObjNum 580 snap 138 seq_num 580) dirty exists 2024-03-27T09:39:55.236 INFO:tasks.rados.rados.0.smithi170.stdout:1204: left oid 8 (ObjNum 580 snap 138 seq_num 580) 2024-03-27T09:39:55.236 INFO:tasks.rados.rados.0.smithi170.stdout:1209: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-03-27T09:39:55.244 INFO:tasks.rados.rados.0.smithi170.stdout:1209: writing smithi17031214-10 from 598481 to 1311476 tid 1 2024-03-27T09:39:55.249 INFO:tasks.rados.rados.0.smithi170.stdout:1209: writing smithi17031214-10 from 1899355 to 2318220 tid 2 2024-03-27T09:39:55.257 INFO:tasks.rados.rados.0.smithi170.stdout:1209: writing smithi17031214-10 from 3093840 to 3798481 tid 3 2024-03-27T09:39:55.258 INFO:tasks.rados.rados.0.smithi170.stdout:1204: done (4 left) 2024-03-27T09:39:55.258 INFO:tasks.rados.rados.0.smithi170.stdout:1210: read oid 33 snap 137 2024-03-27T09:39:55.258 INFO:tasks.rados.rados.0.smithi170.stdout:1210: expect (ObjNum 576 snap 137 seq_num 576) 2024-03-27T09:39:55.258 INFO:tasks.rados.rados.0.smithi170.stdout:1211: snap_create 2024-03-27T09:39:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:55.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:55.280 INFO:tasks.rados.rados.0.smithi170.stdout:1206: finishing write tid 1 to smithi17031214-40 2024-03-27T09:39:55.288 INFO:tasks.rados.rados.0.smithi170.stdout:1210: expect (ObjNum 576 snap 137 seq_num 576) 2024-03-27T09:39:55.556 INFO:tasks.rados.rados.0.smithi170.stdout:1208: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:55.719 INFO:tasks.rados.rados.0.smithi170.stdout:1209: finishing write tid 1 to smithi17031214-10 2024-03-27T09:39:55.719 INFO:tasks.rados.rados.0.smithi170.stdout:1206: finishing write tid 2 to smithi17031214-40 2024-03-27T09:39:55.719 INFO:tasks.rados.rados.0.smithi170.stdout:1209: finishing write tid 2 to smithi17031214-10 2024-03-27T09:39:55.719 INFO:tasks.rados.rados.0.smithi170.stdout:1206: finishing write tid 3 to smithi17031214-40 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: finishing write tid 4 to smithi17031214-40 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: finishing write tid 5 to smithi17031214-40 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: oid 40 updating version 0 to 488 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: oid 40 version 488 is already newer than 487 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: oid 40 updating version 488 to 489 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: oid 40 updating version 489 to 490 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 490 (ObjNum 582 snap 138 seq_num 582) dirty exists 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1206: left oid 40 (ObjNum 582 snap 138 seq_num 582) 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: finishing write tid 3 to smithi17031214-10 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: finishing write tid 4 to smithi17031214-10 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: finishing write tid 5 to smithi17031214-10 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: oid 10 updating version 0 to 52 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: oid 10 updating version 52 to 53 2024-03-27T09:39:55.720 INFO:tasks.rados.rados.0.smithi170.stdout:1209: oid 10 version 53 is already newer than 51 2024-03-27T09:39:55.721 INFO:tasks.rados.rados.0.smithi170.stdout:1209: oid 10 updating version 53 to 54 2024-03-27T09:39:55.721 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 54 (ObjNum 583 snap 138 seq_num 583) dirty exists 2024-03-27T09:39:55.721 INFO:tasks.rados.rados.0.smithi170.stdout:1209: left oid 10 (ObjNum 583 snap 138 seq_num 583) 2024-03-27T09:39:55.721 INFO:tasks.rados.rados.0.smithi170.stdout:1207: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:39:55.985 INFO:tasks.rados.rados.0.smithi170.stdout:1206: done (5 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1207: done (4 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1208: done (3 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1209: done (2 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1210: done (1 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1211: done (0 left) 2024-03-27T09:39:55.986 INFO:tasks.rados.rados.0.smithi170.stdout:1212: delete oid 41 current snap is 139 2024-03-27T09:39:56.046 INFO:tasks.rados.rados.0.smithi170.stdout:1212: done (0 left) 2024-03-27T09:39:56.046 INFO:tasks.rados.rados.0.smithi170.stdout:1213: read oid 43 snap -1 2024-03-27T09:39:56.047 INFO:tasks.rados.rados.0.smithi170.stdout:1213: expect (ObjNum 537 snap 124 seq_num 537) 2024-03-27T09:39:56.047 INFO:tasks.rados.rados.0.smithi170.stdout:1214: delete oid 18 current snap is 139 2024-03-27T09:39:56.071 INFO:tasks.rados.rados.0.smithi170.stdout:1214: done (1 left) 2024-03-27T09:39:56.071 INFO:tasks.rados.rados.0.smithi170.stdout:1215: read oid 17 snap 138 2024-03-27T09:39:56.071 INFO:tasks.rados.rados.0.smithi170.stdout:1215: expect (ObjNum 574 snap 137 seq_num 574) 2024-03-27T09:39:56.071 INFO:tasks.rados.rados.0.smithi170.stdout:1216: write oid 10 current snap is 139 2024-03-27T09:39:56.072 INFO:tasks.rados.rados.0.smithi170.stdout:1216: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-03-27T09:39:56.078 INFO:tasks.rados.rados.0.smithi170.stdout:1216: writing smithi17031214-10 from 648902 to 1303997 tid 1 2024-03-27T09:39:56.086 INFO:tasks.rados.rados.0.smithi170.stdout:1216: writing smithi17031214-10 from 2074801 to 2640993 tid 2 2024-03-27T09:39:56.093 INFO:tasks.rados.rados.0.smithi170.stdout:1216: writing smithi17031214-10 from 3176109 to 3848902 tid 3 2024-03-27T09:39:56.093 INFO:tasks.rados.rados.0.smithi170.stdout:1217: snap_create 2024-03-27T09:39:56.135 INFO:tasks.rados.rados.0.smithi170.stdout:1215: expect (ObjNum 574 snap 137 seq_num 574) 2024-03-27T09:39:56.277 INFO:tasks.rados.rados.0.smithi170.stdout:1216: finishing write tid 1 to smithi17031214-10 2024-03-27T09:39:56.277 INFO:tasks.rados.rados.0.smithi170.stdout:1213: expect (ObjNum 537 snap 124 seq_num 537) 2024-03-27T09:39:56.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:39:56.476 INFO:tasks.rados.rados.0.smithi170.stdout:1216: finishing write tid 2 to smithi17031214-10 2024-03-27T09:39:56.476 INFO:tasks.rados.rados.0.smithi170.stdout:1216: finishing write tid 3 to smithi17031214-10 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: finishing write tid 4 to smithi17031214-10 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: finishing write tid 5 to smithi17031214-10 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: oid 10 updating version 0 to 59 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: oid 10 version 59 is already newer than 56 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: oid 10 version 59 is already newer than 57 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: oid 10 version 59 is already newer than 59 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 59 (ObjNum 586 snap 139 seq_num 586) dirty exists 2024-03-27T09:39:56.477 INFO:tasks.rados.rados.0.smithi170.stdout:1216: left oid 10 (ObjNum 586 snap 139 seq_num 586) 2024-03-27T09:39:56.525 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:39:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:39:56.528 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1213: done (3 left) 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1215: done (2 left) 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1216: done (1 left) 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1217: done (0 left) 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1218: read oid 28 snap 137 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1218: expect deleted 2024-03-27T09:39:57.015 INFO:tasks.rados.rados.0.smithi170.stdout:1219: snap_remove snap 138 2024-03-27T09:39:57.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:39:57.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:57.376+0000 7f75303d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:57.382 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:57.381+0000 7f75303d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:57.405 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:57.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1219: done (1 left) 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1220: read oid 46 snap -1 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1220: expect deleted 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1221: read oid 14 snap -1 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1221: expect deleted 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1222: read oid 29 snap 134 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1222: expect deleted 2024-03-27T09:39:58.019 INFO:tasks.rados.rados.0.smithi170.stdout:1223: delete oid 1 current snap is 140 2024-03-27T09:39:58.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~248s 2024-03-27T09:39:58.125 INFO:tasks.rados.rados.0.smithi170.stdout:1218: done (4 left) 2024-03-27T09:39:58.125 INFO:tasks.rados.rados.0.smithi170.stdout:1220: done (3 left) 2024-03-27T09:39:58.125 INFO:tasks.rados.rados.0.smithi170.stdout:1221: done (2 left) 2024-03-27T09:39:58.126 INFO:tasks.rados.rados.0.smithi170.stdout:1222: done (1 left) 2024-03-27T09:39:58.126 INFO:tasks.rados.rados.0.smithi170.stdout:1223: done (0 left) 2024-03-27T09:39:58.126 INFO:tasks.rados.rados.0.smithi170.stdout:1224: write oid 22 current snap is 140 2024-03-27T09:39:58.126 INFO:tasks.rados.rados.0.smithi170.stdout:1224: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-03-27T09:39:58.131 INFO:tasks.rados.rados.0.smithi170.stdout:1224: writing smithi17031214-22 from 682516 to 1088109 tid 1 2024-03-27T09:39:58.138 INFO:tasks.rados.rados.0.smithi170.stdout:1224: writing smithi17031214-22 from 1791765 to 2361626 tid 2 2024-03-27T09:39:58.144 INFO:tasks.rados.rados.0.smithi170.stdout:1224: writing smithi17031214-22 from 2830955 to 3552075 tid 3 2024-03-27T09:39:58.145 INFO:tasks.rados.rados.0.smithi170.stdout:1225: write oid 21 current snap is 140 2024-03-27T09:39:58.145 INFO:tasks.rados.rados.0.smithi170.stdout:1225: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-03-27T09:39:58.150 INFO:tasks.rados.rados.0.smithi170.stdout:1225: writing smithi17031214-21 from 699323 to 1180165 tid 1 2024-03-27T09:39:58.156 INFO:tasks.rados.rados.0.smithi170.stdout:1225: writing smithi17031214-21 from 1850247 to 2280119 tid 2 2024-03-27T09:39:58.162 INFO:tasks.rados.rados.0.smithi170.stdout:1225: writing smithi17031214-21 from 2858378 to 3390770 tid 3 2024-03-27T09:39:58.163 INFO:tasks.rados.rados.0.smithi170.stdout:1226: snap_create 2024-03-27T09:39:58.166 INFO:tasks.rados.rados.0.smithi170.stdout:1224: finishing write tid 1 to smithi17031214-22 2024-03-27T09:39:58.196 INFO:tasks.rados.rados.0.smithi170.stdout:1225: finishing write tid 1 to smithi17031214-21 2024-03-27T09:39:58.210 INFO:tasks.rados.rados.0.smithi170.stdout:1224: finishing write tid 2 to smithi17031214-22 2024-03-27T09:39:58.216 INFO:tasks.rados.rados.0.smithi170.stdout:1225: finishing write tid 2 to smithi17031214-21 2024-03-27T09:39:58.228 INFO:tasks.rados.rados.0.smithi170.stdout:1225: finishing write tid 3 to smithi17031214-21 2024-03-27T09:39:58.234 INFO:tasks.rados.rados.0.smithi170.stdout:1224: finishing write tid 3 to smithi17031214-22 2024-03-27T09:39:58.237 INFO:tasks.rados.rados.0.smithi170.stdout:1224: finishing write tid 4 to smithi17031214-22 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: finishing write tid 5 to smithi17031214-22 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: oid 22 updating version 0 to 217 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: oid 22 updating version 217 to 219 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: oid 22 version 219 is already newer than 216 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: oid 22 version 219 is already newer than 219 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 219 (ObjNum 588 snap 140 seq_num 588) dirty exists 2024-03-27T09:39:58.240 INFO:tasks.rados.rados.0.smithi170.stdout:1224: left oid 22 (ObjNum 588 snap 140 seq_num 588) 2024-03-27T09:39:58.241 INFO:tasks.rados.rados.0.smithi170.stdout:1225: finishing write tid 4 to smithi17031214-21 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: finishing write tid 5 to smithi17031214-21 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: oid 21 updating version 0 to 310 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: oid 21 updating version 310 to 313 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: oid 21 version 313 is already newer than 312 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: oid 21 version 313 is already newer than 312 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 313 (ObjNum 589 snap 140 seq_num 589) dirty exists 2024-03-27T09:39:58.244 INFO:tasks.rados.rados.0.smithi170.stdout:1225: left oid 21 (ObjNum 589 snap 140 seq_num 589) 2024-03-27T09:39:59.019 INFO:tasks.rados.rados.0.smithi170.stdout:1224: done (2 left) 2024-03-27T09:39:59.019 INFO:tasks.rados.rados.0.smithi170.stdout:1225: done (1 left) 2024-03-27T09:39:59.019 INFO:tasks.rados.rados.0.smithi170.stdout:1226: done (0 left) 2024-03-27T09:39:59.019 INFO:tasks.rados.rados.0.smithi170.stdout:1227: write oid 32 current snap is 141 2024-03-27T09:39:59.020 INFO:tasks.rados.rados.0.smithi170.stdout:1227: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-03-27T09:39:59.026 INFO:tasks.rados.rados.0.smithi170.stdout:1227: writing smithi17031214-32 from 716130 to 1272221 tid 1 2024-03-27T09:39:59.033 INFO:tasks.rados.rados.0.smithi170.stdout:1227: writing smithi17031214-32 from 1792376 to 2365906 tid 2 2024-03-27T09:39:59.038 INFO:tasks.rados.rados.0.smithi170.stdout:1227: writing smithi17031214-32 from 2769448 to 3229465 tid 3 2024-03-27T09:39:59.042 INFO:tasks.rados.rados.0.smithi170.stdout:1227: writing smithi17031214-32 from 3821956 to 3916130 tid 4 2024-03-27T09:39:59.042 INFO:tasks.rados.rados.0.smithi170.stdout:1228: delete oid 34 current snap is 141 2024-03-27T09:39:59.072 INFO:tasks.rados.rados.0.smithi170.stdout:1228: done (1 left) 2024-03-27T09:39:59.073 INFO:tasks.rados.rados.0.smithi170.stdout:1229: read oid 20 snap -1 2024-03-27T09:39:59.073 INFO:tasks.rados.rados.0.smithi170.stdout:1229: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:59.073 INFO:tasks.rados.rados.0.smithi170.stdout:1230: delete oid 36 current snap is 141 2024-03-27T09:39:59.100 INFO:tasks.rados.rados.0.smithi170.stdout:1230: done (2 left) 2024-03-27T09:39:59.100 INFO:tasks.rados.rados.0.smithi170.stdout:1231: snap_create 2024-03-27T09:39:59.109 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 1 to smithi17031214-32 2024-03-27T09:39:59.112 INFO:tasks.rados.rados.0.smithi170.stdout:1229: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:39:59.276 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 2 to smithi17031214-32 2024-03-27T09:39:59.276 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 3 to smithi17031214-32 2024-03-27T09:39:59.276 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 4 to smithi17031214-32 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 5 to smithi17031214-32 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: finishing write tid 6 to smithi17031214-32 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: oid 32 updating version 0 to 393 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: oid 32 version 393 is already newer than 391 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: oid 32 version 393 is already newer than 391 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: oid 32 updating version 393 to 394 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: oid 32 version 394 is already newer than 392 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 394 (ObjNum 590 snap 141 seq_num 590) dirty exists 2024-03-27T09:39:59.277 INFO:tasks.rados.rados.0.smithi170.stdout:1227: left oid 32 (ObjNum 590 snap 141 seq_num 590) 2024-03-27T09:39:59.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:39:59.562 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:59.560+0000 7fde1626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:59.566 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:39:59.565+0000 7fde1626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:39:59.589 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:39:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:39:59.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:00.024 INFO:tasks.rados.rados.0.smithi170.stdout:1227: done (2 left) 2024-03-27T09:40:00.024 INFO:tasks.rados.rados.0.smithi170.stdout:1229: done (1 left) 2024-03-27T09:40:00.024 INFO:tasks.rados.rados.0.smithi170.stdout:1231: done (0 left) 2024-03-27T09:40:00.024 INFO:tasks.rados.rados.0.smithi170.stdout:1232: delete oid 15 current snap is 142 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1232: done (0 left) 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1233: read oid 32 snap -1 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1233: expect (ObjNum 590 snap 141 seq_num 590) 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1234: read oid 49 snap -1 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1234: expect (ObjNum 509 snap 116 seq_num 509) 2024-03-27T09:40:00.095 INFO:tasks.rados.rados.0.smithi170.stdout:1235: read oid 37 snap 137 2024-03-27T09:40:00.096 INFO:tasks.rados.rados.0.smithi170.stdout:1235: expect deleted 2024-03-27T09:40:00.096 INFO:tasks.rados.rados.0.smithi170.stdout:1236: write (excl) oid 28 current snap is 142 2024-03-27T09:40:00.096 INFO:tasks.rados.rados.0.smithi170.stdout:1236: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-03-27T09:40:00.103 INFO:tasks.rados.rados.0.smithi170.stdout:1236: writing smithi17031214-28 from 783358 to 1356798 tid 1 2024-03-27T09:40:00.110 INFO:tasks.rados.rados.0.smithi170.stdout:1236: writing smithi17031214-28 from 2026304 to 2607172 tid 2 2024-03-27T09:40:00.118 INFO:tasks.rados.rados.0.smithi170.stdout:1236: writing smithi17031214-28 from 3279140 to 3900598 tid 3 2024-03-27T09:40:00.118 INFO:tasks.rados.rados.0.smithi170.stdout:1237: read oid 34 snap -1 2024-03-27T09:40:00.118 INFO:tasks.rados.rados.0.smithi170.stdout:1237: expect deleted 2024-03-27T09:40:00.118 INFO:tasks.rados.rados.0.smithi170.stdout:1238: write oid 40 current snap is 142 2024-03-27T09:40:00.118 INFO:tasks.rados.rados.0.smithi170.stdout:1238: seq_num 595 ranges {400165=648689,1684786=315379} 2024-03-27T09:40:00.123 INFO:tasks.rados.rados.0.smithi170.stdout:1238: writing smithi17031214-40 from 400165 to 1048854 tid 1 2024-03-27T09:40:00.129 INFO:tasks.rados.rados.0.smithi170.stdout:1238: writing smithi17031214-40 from 1684786 to 2000165 tid 2 2024-03-27T09:40:00.129 INFO:tasks.rados.rados.0.smithi170.stdout:1239: write (excl) oid 48 current snap is 142 2024-03-27T09:40:00.129 INFO:tasks.rados.rados.0.smithi170.stdout:1239: seq_num 596 ranges {416972=723938,1743268=273704} 2024-03-27T09:40:00.135 INFO:tasks.rados.rados.0.smithi170.stdout:1239: writing smithi17031214-48 from 416972 to 1140910 tid 1 2024-03-27T09:40:00.140 INFO:tasks.rados.rados.0.smithi170.stdout:1239: writing smithi17031214-48 from 1743268 to 2016972 tid 2 2024-03-27T09:40:00.140 INFO:tasks.rados.rados.0.smithi170.stdout:1240: snap_remove snap 134 2024-03-27T09:40:01.027 INFO:tasks.rados.rados.0.smithi170.stdout:1240: done (7 left) 2024-03-27T09:40:01.027 INFO:tasks.rados.rados.0.smithi170.stdout:1241: snap_remove snap 140 2024-03-27T09:40:01.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:01.631 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:01.634 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:01.636 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:01.745 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:01.742+0000 7f56499f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:01.749 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:01.749+0000 7f56499f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:01.774 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:01.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:02.031 INFO:tasks.rados.rados.0.smithi170.stdout:1241: done (7 left) 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1242: read oid 50 snap 141 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1242: expect deleted 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1243: read oid 24 snap 139 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1243: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1244: snap_create 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1238: finishing write tid 1 to smithi17031214-40 2024-03-27T09:40:02.032 INFO:tasks.rados.rados.0.smithi170.stdout:1234: expect (ObjNum 509 snap 116 seq_num 509) 2024-03-27T09:40:02.149 INFO:tasks.rados.rados.0.smithi170.stdout:1233: expect (ObjNum 590 snap 141 seq_num 590) 2024-03-27T09:40:02.402 INFO:tasks.rados.rados.0.smithi170.stdout:1238: finishing write tid 2 to smithi17031214-40 2024-03-27T09:40:02.402 INFO:tasks.rados.rados.0.smithi170.stdout:1236: finishing write tid 1 to smithi17031214-28 2024-03-27T09:40:02.402 INFO:tasks.rados.rados.0.smithi170.stdout:1238: finishing write tid 3 to smithi17031214-40 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1238: finishing write tid 4 to smithi17031214-40 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1238: oid 40 updating version 0 to 493 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1238: oid 40 updating version 493 to 495 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1238: oid 40 version 495 is already newer than 494 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 495 (ObjNum 595 snap 142 seq_num 595) dirty exists 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1238: left oid 40 (ObjNum 595 snap 142 seq_num 595) 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1239: finishing write tid 1 to smithi17031214-48 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1236: finishing write tid 2 to smithi17031214-28 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1239: finishing write tid 2 to smithi17031214-48 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1236: finishing write tid 3 to smithi17031214-28 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1236: finishing write tid 4 to smithi17031214-28 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1239: finishing write tid 3 to smithi17031214-48 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1236: finishing write tid 5 to smithi17031214-28 2024-03-27T09:40:02.403 INFO:tasks.rados.rados.0.smithi170.stdout:1236: oid 28 updating version 0 to 397 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1236: oid 28 updating version 397 to 398 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1236: oid 28 version 398 is already newer than 395 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1236: oid 28 version 398 is already newer than 396 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 398 (ObjNum 594 snap 142 seq_num 594) dirty exists 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1236: left oid 28 (ObjNum 594 snap 142 seq_num 594) 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1239: finishing write tid 4 to smithi17031214-48 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1239: oid 48 updating version 0 to 308 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1239: oid 48 version 308 is already newer than 307 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1239: oid 48 version 308 is already newer than 306 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 308 (ObjNum 596 snap 142 seq_num 596) dirty exists 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1239: left oid 48 (ObjNum 596 snap 142 seq_num 596) 2024-03-27T09:40:02.404 INFO:tasks.rados.rados.0.smithi170.stdout:1243: expect (ObjNum 442 snap 103 seq_num 442) 2024-03-27T09:40:03.033 INFO:tasks.rados.rados.0.smithi170.stdout:1233: done (9 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1234: done (8 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1235: done (7 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1236: done (6 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1237: done (5 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1238: done (4 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1239: done (3 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1242: done (2 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1243: done (1 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1244: done (0 left) 2024-03-27T09:40:03.034 INFO:tasks.rados.rados.0.smithi170.stdout:1245: snap_create 2024-03-27T09:40:03.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~253s 2024-03-27T09:40:03.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:03.934 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:03.932+0000 7f71bfe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:03.937 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:03.937+0000 7f71bfe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:03.961 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:04.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:04.034 INFO:tasks.rados.rados.0.smithi170.stdout:1245: done (0 left) 2024-03-27T09:40:04.035 INFO:tasks.rados.rados.0.smithi170.stdout:1246: snap_remove snap 142 2024-03-27T09:40:05.039 INFO:tasks.rados.rados.0.smithi170.stdout:1246: done (0 left) 2024-03-27T09:40:05.039 INFO:tasks.rados.rados.0.smithi170.stdout:1247: delete oid 3 current snap is 144 2024-03-27T09:40:05.122 INFO:tasks.rados.rados.0.smithi170.stdout:1247: done (0 left) 2024-03-27T09:40:05.122 INFO:tasks.rados.rados.0.smithi170.stdout:1248: write oid 14 current snap is 144 2024-03-27T09:40:05.122 INFO:tasks.rados.rados.0.smithi170.stdout:1248: seq_num 598 ranges {450586=474436,1343879=588206} 2024-03-27T09:40:05.127 INFO:tasks.rados.rados.0.smithi170.stdout:1248: writing smithi17031214-14 from 450586 to 925022 tid 1 2024-03-27T09:40:05.133 INFO:tasks.rados.rados.0.smithi170.stdout:1248: writing smithi17031214-14 from 1343879 to 1932085 tid 2 2024-03-27T09:40:05.133 INFO:tasks.rados.rados.0.smithi170.stdout:1249: read oid 27 snap 139 2024-03-27T09:40:05.133 INFO:tasks.rados.rados.0.smithi170.stdout:1249: expect (ObjNum 558 snap 131 seq_num 558) 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1250: read oid 13 snap -1 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1250: expect deleted 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1251: read oid 21 snap -1 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1251: expect (ObjNum 589 snap 140 seq_num 589) 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1252: write oid 9 current snap is 144 2024-03-27T09:40:05.134 INFO:tasks.rados.rados.0.smithi170.stdout:1252: seq_num 599 ranges {467393=549685,1802361=265032} 2024-03-27T09:40:05.140 INFO:tasks.rados.rados.0.smithi170.stdout:1252: writing smithi17031214-9 from 467393 to 1017078 tid 1 2024-03-27T09:40:05.146 INFO:tasks.rados.rados.0.smithi170.stdout:1252: writing smithi17031214-9 from 1802361 to 2067393 tid 2 2024-03-27T09:40:05.146 INFO:tasks.rados.rados.0.smithi170.stdout:1253: read oid 17 snap 137 2024-03-27T09:40:05.146 INFO:tasks.rados.rados.0.smithi170.stdout:1253: expect (ObjNum 574 snap 137 seq_num 574) 2024-03-27T09:40:05.146 INFO:tasks.rados.rados.0.smithi170.stdout:1254: read oid 28 snap -1 2024-03-27T09:40:05.147 INFO:tasks.rados.rados.0.smithi170.stdout:1254: expect (ObjNum 594 snap 142 seq_num 594) 2024-03-27T09:40:05.147 INFO:tasks.rados.rados.0.smithi170.stdout:1255: delete oid 25 current snap is 144 2024-03-27T09:40:05.160 INFO:tasks.rados.rados.0.smithi170.stdout:1248: finishing write tid 1 to smithi17031214-14 2024-03-27T09:40:05.199 INFO:tasks.rados.rados.0.smithi170.stdout:1254: expect (ObjNum 594 snap 142 seq_num 594) 2024-03-27T09:40:05.362 INFO:tasks.rados.rados.0.smithi170.stdout:1248: finishing write tid 2 to smithi17031214-14 2024-03-27T09:40:05.362 INFO:tasks.rados.rados.0.smithi170.stdout:1252: finishing write tid 1 to smithi17031214-9 2024-03-27T09:40:05.362 INFO:tasks.rados.rados.0.smithi170.stdout:1251: expect (ObjNum 589 snap 140 seq_num 589) 2024-03-27T09:40:05.558 INFO:tasks.rados.rados.0.smithi170.stdout:1253: expect (ObjNum 574 snap 137 seq_num 574) 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:1248: finishing write tid 3 to smithi17031214-14 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:1248: finishing write tid 4 to smithi17031214-14 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:1248: oid 14 updating version 0 to 263 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:1248: oid 14 version 263 is already newer than 262 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:1248: oid 14 updating version 263 to 264 2024-03-27T09:40:05.692 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 264 (ObjNum 598 snap 144 seq_num 598) dirty exists 2024-03-27T09:40:05.693 INFO:tasks.rados.rados.0.smithi170.stdout:1248: left oid 14 (ObjNum 598 snap 144 seq_num 598) 2024-03-27T09:40:05.693 INFO:tasks.rados.rados.0.smithi170.stdout:1249: expect (ObjNum 558 snap 131 seq_num 558) 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: finishing write tid 2 to smithi17031214-9 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: finishing write tid 3 to smithi17031214-9 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: finishing write tid 4 to smithi17031214-9 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: oid 9 updating version 0 to 503 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: oid 9 version 503 is already newer than 502 2024-03-27T09:40:05.889 INFO:tasks.rados.rados.0.smithi170.stdout:1252: oid 9 version 503 is already newer than 501 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 503 (ObjNum 599 snap 144 seq_num 599) dirty exists 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1252: left oid 9 (ObjNum 599 snap 144 seq_num 599) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1248: done (7 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1249: done (6 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1250: done (5 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1251: done (4 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1252: done (3 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1253: done (2 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1254: done (1 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1255: done (0 left) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1256: read oid 26 snap -1 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1256: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:40:05.890 INFO:tasks.rados.rados.0.smithi170.stdout:1257: snap_create 2024-03-27T09:40:05.908 INFO:tasks.rados.rados.0.smithi170.stdout:1256: expect (ObjNum 449 snap 104 seq_num 449) 2024-03-27T09:40:06.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:06.123 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:06.120+0000 7f701402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:06.126 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:06.126+0000 7f701402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:06.134 INFO:tasks.rados.rados.0.smithi170.stdout:1256: done (1 left) 2024-03-27T09:40:06.134 INFO:tasks.rados.rados.0.smithi170.stdout:1257: done (0 left) 2024-03-27T09:40:06.134 INFO:tasks.rados.rados.0.smithi170.stdout:1258: write (excl) oid 40 current snap is 145 2024-03-27T09:40:06.134 INFO:tasks.rados.rados.0.smithi170.stdout:1258: seq_num 601 ranges {501007=416536,1635678=465329} 2024-03-27T09:40:06.139 INFO:tasks.rados.rados.0.smithi170.stdout:1258: writing smithi17031214-40 from 501007 to 917543 tid 1 2024-03-27T09:40:06.146 INFO:tasks.rados.rados.0.smithi170.stdout:1258: writing smithi17031214-40 from 1635678 to 2101007 tid 2 2024-03-27T09:40:06.146 INFO:tasks.rados.rados.0.smithi170.stdout:1259: write (excl) oid 4 current snap is 145 2024-03-27T09:40:06.146 INFO:tasks.rados.rados.0.smithi170.stdout:1259: seq_num 602 ranges {517814=491785,1577807=540007} 2024-03-27T09:40:06.151 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:06.152 INFO:tasks.rados.rados.0.smithi170.stdout:1259: writing smithi17031214-4 from 517814 to 1009599 tid 1 2024-03-27T09:40:06.158 INFO:tasks.rados.rados.0.smithi170.stdout:1259: writing smithi17031214-4 from 1577807 to 2117814 tid 2 2024-03-27T09:40:06.158 INFO:tasks.rados.rados.0.smithi170.stdout:1260: write oid 30 current snap is 145 2024-03-27T09:40:06.158 INFO:tasks.rados.rados.0.smithi170.stdout:1260: seq_num 603 ranges {534621=567034,1636289=498332} 2024-03-27T09:40:06.164 INFO:tasks.rados.rados.0.smithi170.stdout:1260: writing smithi17031214-30 from 534621 to 1101655 tid 1 2024-03-27T09:40:06.170 INFO:tasks.rados.rados.0.smithi170.stdout:1260: writing smithi17031214-30 from 1636289 to 2134621 tid 2 2024-03-27T09:40:06.170 INFO:tasks.rados.rados.0.smithi170.stdout:1261: read oid 44 snap -1 2024-03-27T09:40:06.170 INFO:tasks.rados.rados.0.smithi170.stdout:1261: expect deleted 2024-03-27T09:40:06.170 INFO:tasks.rados.rados.0.smithi170.stdout:1262: write oid 41 current snap is 145 2024-03-27T09:40:06.170 INFO:tasks.rados.rados.0.smithi170.stdout:1262: seq_num 604 ranges {551428=642283,1694771=456657} 2024-03-27T09:40:06.177 INFO:tasks.rados.rados.0.smithi170.stdout:1262: writing smithi17031214-41 from 551428 to 1193711 tid 1 2024-03-27T09:40:06.182 INFO:tasks.rados.rados.0.smithi170.stdout:1262: writing smithi17031214-41 from 1694771 to 2151428 tid 2 2024-03-27T09:40:06.182 INFO:tasks.rados.rados.0.smithi170.stdout:1263: write oid 29 current snap is 145 2024-03-27T09:40:06.182 INFO:tasks.rados.rados.0.smithi170.stdout:1263: seq_num 605 ranges {568235=717532,1753253=414982} 2024-03-27T09:40:06.189 INFO:tasks.rados.rados.0.smithi170.stdout:1263: writing smithi17031214-29 from 568235 to 1285767 tid 1 2024-03-27T09:40:06.194 INFO:tasks.rados.rados.0.smithi170.stdout:1263: writing smithi17031214-29 from 1753253 to 2168235 tid 2 2024-03-27T09:40:06.194 INFO:tasks.rados.rados.0.smithi170.stdout:1264: write (excl) oid 1 current snap is 145 2024-03-27T09:40:06.195 INFO:tasks.rados.rados.0.smithi170.stdout:1264: seq_num 606 ranges {585042=792781,2095382=89660} 2024-03-27T09:40:06.202 INFO:tasks.rados.rados.0.smithi170.stdout:1264: writing smithi17031214-1 from 585042 to 1377823 tid 1 2024-03-27T09:40:06.207 INFO:tasks.rados.rados.0.smithi170.stdout:1264: writing smithi17031214-1 from 2095382 to 2185042 tid 2 2024-03-27T09:40:06.207 INFO:tasks.rados.rados.0.smithi170.stdout:1265: snap_remove snap 139 2024-03-27T09:40:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:06.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:06.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:06.735 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:06.739 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:07.051 INFO:tasks.rados.rados.0.smithi170.stdout:1265: done (7 left) 2024-03-27T09:40:07.051 INFO:tasks.rados.rados.0.smithi170.stdout:1266: write oid 26 current snap is 145 2024-03-27T09:40:07.051 INFO:tasks.rados.rados.0.smithi170.stdout:1266: seq_num 607 ranges {601849=468030,1753864=447985} 2024-03-27T09:40:07.053 INFO:tasks.rados.rados.0.smithi170.stdout:1266: writing smithi17031214-26 from 601849 to 1069879 tid 1 2024-03-27T09:40:07.059 INFO:tasks.rados.rados.0.smithi170.stdout:1266: writing smithi17031214-26 from 1753864 to 2201849 tid 2 2024-03-27T09:40:07.059 INFO:tasks.rados.rados.0.smithi170.stdout:1267: write (excl) oid 31 current snap is 145 2024-03-27T09:40:07.059 INFO:tasks.rados.rados.0.smithi170.stdout:1267: seq_num 608 ranges {618656=543279,1812346=406310} 2024-03-27T09:40:07.065 INFO:tasks.rados.rados.0.smithi170.stdout:1267: writing smithi17031214-31 from 618656 to 1161935 tid 1 2024-03-27T09:40:07.072 INFO:tasks.rados.rados.0.smithi170.stdout:1267: writing smithi17031214-31 from 1812346 to 2218656 tid 2 2024-03-27T09:40:07.072 INFO:tasks.rados.rados.0.smithi170.stdout:1268: read oid 39 snap -1 2024-03-27T09:40:07.072 INFO:tasks.rados.rados.0.smithi170.stdout:1268: expect (ObjNum 567 snap 135 seq_num 567) 2024-03-27T09:40:07.072 INFO:tasks.rados.rados.0.smithi170.stdout:1269: write oid 27 current snap is 145 2024-03-27T09:40:07.072 INFO:tasks.rados.rados.0.smithi170.stdout:1269: seq_num 609 ranges {635463=734881,1987181=248282} 2024-03-27T09:40:07.080 INFO:tasks.rados.rados.0.smithi170.stdout:1269: writing smithi17031214-27 from 635463 to 1370344 tid 1 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1269: writing smithi17031214-27 from 1987181 to 2235463 tid 2 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1270: snap_remove snap 144 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: finishing write tid 1 to smithi17031214-41 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: finishing write tid 2 to smithi17031214-41 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: finishing write tid 3 to smithi17031214-41 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1263: finishing write tid 1 to smithi17031214-29 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: finishing write tid 4 to smithi17031214-41 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: oid 41 updating version 0 to 411 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: oid 41 version 411 is already newer than 409 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: oid 41 version 411 is already newer than 410 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 411 (ObjNum 604 snap 145 seq_num 604) dirty exists 2024-03-27T09:40:07.086 INFO:tasks.rados.rados.0.smithi170.stdout:1262: left oid 41 (ObjNum 604 snap 145 seq_num 604) 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1259: finishing write tid 1 to smithi17031214-4 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1258: finishing write tid 1 to smithi17031214-40 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1263: finishing write tid 2 to smithi17031214-29 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1260: finishing write tid 1 to smithi17031214-30 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1264: finishing write tid 1 to smithi17031214-1 2024-03-27T09:40:07.087 INFO:tasks.rados.rados.0.smithi170.stdout:1259: finishing write tid 2 to smithi17031214-4 2024-03-27T09:40:08.052 INFO:tasks.rados.rados.0.smithi170.stdout:1261: done (11 left) 2024-03-27T09:40:08.052 INFO:tasks.rados.rados.0.smithi170.stdout:1262: done (10 left) 2024-03-27T09:40:08.052 INFO:tasks.rados.rados.0.smithi170.stdout:1270: done (9 left) 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1271: delete oid 6 current snap is 145 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1264: finishing write tid 2 to smithi17031214-1 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: finishing write tid 3 to smithi17031214-29 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1258: finishing write tid 2 to smithi17031214-40 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: finishing write tid 4 to smithi17031214-29 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: oid 29 updating version 0 to 375 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: oid 29 updating version 375 to 377 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: oid 29 version 377 is already newer than 376 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 377 (ObjNum 605 snap 145 seq_num 605) dirty exists 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1263: left oid 29 (ObjNum 605 snap 145 seq_num 605) 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1259: finishing write tid 3 to smithi17031214-4 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1259: finishing write tid 4 to smithi17031214-4 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1259: oid 4 updating version 0 to 55 2024-03-27T09:40:08.053 INFO:tasks.rados.rados.0.smithi170.stdout:1259: oid 4 updating version 55 to 56 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1259: oid 4 version 56 is already newer than 54 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 56 (ObjNum 602 snap 145 seq_num 602) dirty exists 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1259: left oid 4 (ObjNum 602 snap 145 seq_num 602) 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1260: finishing write tid 2 to smithi17031214-30 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: finishing write tid 3 to smithi17031214-1 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: finishing write tid 4 to smithi17031214-1 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: oid 1 updating version 0 to 228 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: oid 1 version 228 is already newer than 226 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: oid 1 version 228 is already newer than 227 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 228 (ObjNum 606 snap 145 seq_num 606) dirty exists 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1264: left oid 1 (ObjNum 606 snap 145 seq_num 606) 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1258: finishing write tid 3 to smithi17031214-40 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1258: finishing write tid 4 to smithi17031214-40 2024-03-27T09:40:08.054 INFO:tasks.rados.rados.0.smithi170.stdout:1258: oid 40 updating version 0 to 505 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1258: oid 40 updating version 505 to 506 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1258: oid 40 updating version 506 to 507 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 507 (ObjNum 601 snap 145 seq_num 601) dirty exists 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1258: left oid 40 (ObjNum 601 snap 145 seq_num 601) 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: finishing write tid 3 to smithi17031214-30 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: finishing write tid 4 to smithi17031214-30 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: oid 30 updating version 0 to 267 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: oid 30 version 267 is already newer than 266 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: oid 30 version 267 is already newer than 265 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 267 (ObjNum 603 snap 145 seq_num 603) dirty exists 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1260: left oid 30 (ObjNum 603 snap 145 seq_num 603) 2024-03-27T09:40:08.055 INFO:tasks.rados.rados.0.smithi170.stdout:1268: expect (ObjNum 567 snap 135 seq_num 567) 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1267: finishing write tid 1 to smithi17031214-31 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: finishing write tid 1 to smithi17031214-27 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1266: finishing write tid 1 to smithi17031214-26 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: finishing write tid 2 to smithi17031214-27 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1267: finishing write tid 2 to smithi17031214-31 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1266: finishing write tid 2 to smithi17031214-26 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: finishing write tid 3 to smithi17031214-27 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: finishing write tid 4 to smithi17031214-27 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: oid 27 updating version 0 to 428 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: oid 27 version 428 is already newer than 427 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:1269: oid 27 version 428 is already newer than 427 2024-03-27T09:40:08.201 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 428 (ObjNum 609 snap 145 seq_num 609) dirty exists 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1269: left oid 27 (ObjNum 609 snap 145 seq_num 609) 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: finishing write tid 3 to smithi17031214-31 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: finishing write tid 4 to smithi17031214-31 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: oid 31 updating version 0 to 379 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: oid 31 updating version 379 to 380 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: oid 31 updating version 380 to 381 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 381 (ObjNum 608 snap 145 seq_num 608) dirty exists 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1267: left oid 31 (ObjNum 608 snap 145 seq_num 608) 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1266: finishing write tid 3 to smithi17031214-26 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1258: done (9 left) 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1259: done (8 left) 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1260: done (7 left) 2024-03-27T09:40:08.202 INFO:tasks.rados.rados.0.smithi170.stdout:1263: done (6 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1264: done (5 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1267: done (4 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1268: done (3 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1269: done (2 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1271: done (1 left) 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1272: snap_create 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1266: finishing write tid 4 to smithi17031214-26 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1266: oid 26 updating version 0 to 320 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1266: oid 26 version 320 is already newer than 319 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1266: oid 26 updating version 320 to 321 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 321 (ObjNum 607 snap 145 seq_num 607) dirty exists 2024-03-27T09:40:08.203 INFO:tasks.rados.rados.0.smithi170.stdout:1266: left oid 26 (ObjNum 607 snap 145 seq_num 607) 2024-03-27T09:40:08.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:08.347 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:08.345+0000 7f1def954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:08.351 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:08.350+0000 7f1def954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:08.375 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:08.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:09.052 INFO:tasks.rados.rados.0.smithi170.stdout:1266: done (1 left) 2024-03-27T09:40:09.052 INFO:tasks.rados.rados.0.smithi170.stdout:1272: done (0 left) 2024-03-27T09:40:09.052 INFO:tasks.rados.rados.0.smithi170.stdout:1273: snap_remove snap 137 2024-03-27T09:40:09.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2024-03-27T09:40:10.059 INFO:tasks.rados.rados.0.smithi170.stdout:1273: done (0 left) 2024-03-27T09:40:10.059 INFO:tasks.rados.rados.0.smithi170.stdout:1274: write (excl) oid 43 current snap is 146 2024-03-27T09:40:10.059 INFO:tasks.rados.rados.0.smithi170.stdout:1274: seq_num 611 ranges {669077=485379,1587792=681285} 2024-03-27T09:40:10.065 INFO:tasks.rados.rados.0.smithi170.stdout:1274: writing smithi17031214-43 from 669077 to 1154456 tid 1 2024-03-27T09:40:10.072 INFO:tasks.rados.rados.0.smithi170.stdout:1274: writing smithi17031214-43 from 1587792 to 2269077 tid 2 2024-03-27T09:40:10.072 INFO:tasks.rados.rados.0.smithi170.stdout:1275: snap_remove snap 141 2024-03-27T09:40:10.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:10.533 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:10.531+0000 7f16d9d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:10.537 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:10.536+0000 7f16d9d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:10.561 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:10.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1275: done (1 left) 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1276: delete oid 28 current snap is 146 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: finishing write tid 1 to smithi17031214-43 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: finishing write tid 2 to smithi17031214-43 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: finishing write tid 3 to smithi17031214-43 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: finishing write tid 4 to smithi17031214-43 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: oid 43 updating version 0 to 436 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: oid 43 updating version 436 to 438 2024-03-27T09:40:11.061 INFO:tasks.rados.rados.0.smithi170.stdout:1274: oid 43 version 438 is already newer than 437 2024-03-27T09:40:11.062 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 438 (ObjNum 611 snap 146 seq_num 611) dirty exists 2024-03-27T09:40:11.062 INFO:tasks.rados.rados.0.smithi170.stdout:1274: left oid 43 (ObjNum 611 snap 146 seq_num 611) 2024-03-27T09:40:11.229 INFO:tasks.rados.rados.0.smithi170.stdout:1274: done (1 left) 2024-03-27T09:40:11.229 INFO:tasks.rados.rados.0.smithi170.stdout:1276: done (0 left) 2024-03-27T09:40:11.229 INFO:tasks.rados.rados.0.smithi170.stdout:1277: delete oid 45 current snap is 146 2024-03-27T09:40:11.299 INFO:tasks.rados.rados.0.smithi170.stdout:1277: done (0 left) 2024-03-27T09:40:11.299 INFO:tasks.rados.rados.0.smithi170.stdout:1278: read oid 27 snap -1 2024-03-27T09:40:11.299 INFO:tasks.rados.rados.0.smithi170.stdout:1278: expect (ObjNum 609 snap 145 seq_num 609) 2024-03-27T09:40:11.299 INFO:tasks.rados.rados.0.smithi170.stdout:1279: write (excl) oid 30 current snap is 146 2024-03-27T09:40:11.299 INFO:tasks.rados.rados.0.smithi170.stdout:1279: seq_num 614 ranges {719498=711126,2046885=272613} 2024-03-27T09:40:11.306 INFO:tasks.rados.rados.0.smithi170.stdout:1279: writing smithi17031214-30 from 719498 to 1430624 tid 1 2024-03-27T09:40:11.311 INFO:tasks.rados.rados.0.smithi170.stdout:1279: writing smithi17031214-30 from 2046885 to 2319498 tid 2 2024-03-27T09:40:11.311 INFO:tasks.rados.rados.0.smithi170.stdout:1280: delete oid 18 current snap is 146 2024-03-27T09:40:11.333 INFO:tasks.rados.rados.0.smithi170.stdout:1280: done (2 left) 2024-03-27T09:40:11.333 INFO:tasks.rados.rados.0.smithi170.stdout:1281: snap_remove snap 145 2024-03-27T09:40:11.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:11.843 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:11.846 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:12.064 INFO:tasks.rados.rados.0.smithi170.stdout:1281: done (2 left) 2024-03-27T09:40:12.064 INFO:tasks.rados.rados.0.smithi170.stdout:1282: write (excl) oid 44 current snap is 146 2024-03-27T09:40:12.064 INFO:tasks.rados.rados.0.smithi170.stdout:1282: seq_num 616 ranges {753112=577977,1880202=472910} 2024-03-27T09:40:12.071 INFO:tasks.rados.rados.0.smithi170.stdout:1282: writing smithi17031214-44 from 753112 to 1331089 tid 1 2024-03-27T09:40:12.077 INFO:tasks.rados.rados.0.smithi170.stdout:1282: writing smithi17031214-44 from 1880202 to 2353112 tid 2 2024-03-27T09:40:12.077 INFO:tasks.rados.rados.0.smithi170.stdout:1283: snap_remove snap 143 2024-03-27T09:40:12.608 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:12.717 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:12.715+0000 7f8ac726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:12.721 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:12.721+0000 7f8ac726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:12.745 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:12.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:13.068 INFO:tasks.rados.rados.0.smithi170.stdout:1283: done (3 left) 2024-03-27T09:40:13.068 INFO:tasks.rados.rados.0.smithi170.stdout:1284: delete oid 4 current snap is 146 2024-03-27T09:40:13.068 INFO:tasks.rados.rados.0.smithi170.stdout:1278: expect (ObjNum 609 snap 145 seq_num 609) 2024-03-27T09:40:13.223 INFO:tasks.rados.rados.0.smithi170.stdout:1279: finishing write tid 1 to smithi17031214-30 2024-03-27T09:40:13.223 INFO:tasks.rados.rados.0.smithi170.stdout:1279: finishing write tid 2 to smithi17031214-30 2024-03-27T09:40:13.223 INFO:tasks.rados.rados.0.smithi170.stdout:1279: finishing write tid 3 to smithi17031214-30 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1279: finishing write tid 4 to smithi17031214-30 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1279: oid 30 updating version 0 to 273 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1279: oid 30 version 273 is already newer than 272 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1279: oid 30 version 273 is already newer than 271 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 273 (ObjNum 614 snap 146 seq_num 614) dirty exists 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1279: left oid 30 (ObjNum 614 snap 146 seq_num 614) 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: finishing write tid 1 to smithi17031214-44 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: finishing write tid 2 to smithi17031214-44 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: finishing write tid 3 to smithi17031214-44 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: finishing write tid 4 to smithi17031214-44 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: oid 44 updating version 0 to 407 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: oid 44 version 407 is already newer than 405 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:1282: oid 44 version 407 is already newer than 407 2024-03-27T09:40:13.224 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 407 (ObjNum 616 snap 146 seq_num 616) dirty exists 2024-03-27T09:40:13.225 INFO:tasks.rados.rados.0.smithi170.stdout:1282: left oid 44 (ObjNum 616 snap 146 seq_num 616) 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1278: done (3 left) 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1279: done (2 left) 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1282: done (1 left) 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1284: done (0 left) 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1285: write (excl) oid 27 current snap is 146 2024-03-27T09:40:13.238 INFO:tasks.rados.rados.0.smithi170.stdout:1285: seq_num 618 ranges {786726=728475,2280813=105913} 2024-03-27T09:40:13.246 INFO:tasks.rados.rados.0.smithi170.stdout:1285: writing smithi17031214-27 from 786726 to 1515201 tid 1 2024-03-27T09:40:13.251 INFO:tasks.rados.rados.0.smithi170.stdout:1285: writing smithi17031214-27 from 2280813 to 2386726 tid 2 2024-03-27T09:40:13.251 INFO:tasks.rados.rados.0.smithi170.stdout:1286: delete oid 49 current snap is 146 2024-03-27T09:40:13.274 INFO:tasks.rados.rados.0.smithi170.stdout:1286: done (1 left) 2024-03-27T09:40:13.275 INFO:tasks.rados.rados.0.smithi170.stdout:1287: delete oid 21 current snap is 146 2024-03-27T09:40:13.290 INFO:tasks.rados.rados.0.smithi170.stdout:1285: finishing write tid 1 to smithi17031214-27 2024-03-27T09:40:13.308 INFO:tasks.rados.rados.0.smithi170.stdout:1285: finishing write tid 2 to smithi17031214-27 2024-03-27T09:40:13.327 INFO:tasks.rados.rados.0.smithi170.stdout:1285: finishing write tid 3 to smithi17031214-27 2024-03-27T09:40:13.329 INFO:tasks.rados.rados.0.smithi170.stdout:1285: finishing write tid 4 to smithi17031214-27 2024-03-27T09:40:13.329 INFO:tasks.rados.rados.0.smithi170.stdout:1285: oid 27 updating version 0 to 459 2024-03-27T09:40:13.329 INFO:tasks.rados.rados.0.smithi170.stdout:1285: oid 27 updating version 459 to 460 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:1285: oid 27 updating version 460 to 461 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 461 (ObjNum 618 snap 146 seq_num 618) dirty exists 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:1285: left oid 27 (ObjNum 618 snap 146 seq_num 618) 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:1285: done (1 left) 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:1287: done (0 left) 2024-03-27T09:40:13.330 INFO:tasks.rados.rados.0.smithi170.stdout:1288: snap_create 2024-03-27T09:40:14.071 INFO:tasks.rados.rados.0.smithi170.stdout:1288: done (0 left) 2024-03-27T09:40:14.071 INFO:tasks.rados.rados.0.smithi170.stdout:1289: snap_create 2024-03-27T09:40:14.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~264s 2024-03-27T09:40:14.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:14.912 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:14.910+0000 7fc5d88ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:14.916 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:14.916+0000 7fc5d88ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:14.941 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:14.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:15.072 INFO:tasks.rados.rados.0.smithi170.stdout:1289: done (0 left) 2024-03-27T09:40:15.072 INFO:tasks.rados.rados.0.smithi170.stdout:1290: read oid 20 snap -1 2024-03-27T09:40:15.072 INFO:tasks.rados.rados.0.smithi170.stdout:1290: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:40:15.072 INFO:tasks.rados.rados.0.smithi170.stdout:1291: snap_remove snap 146 2024-03-27T09:40:15.990 INFO:tasks.rados.rados.0.smithi170.stdout:1291: done (1 left) 2024-03-27T09:40:15.990 INFO:tasks.rados.rados.0.smithi170.stdout:1292: snap_create 2024-03-27T09:40:15.991 INFO:tasks.rados.rados.0.smithi170.stdout:1290: expect (ObjNum 554 snap 131 seq_num 554) 2024-03-27T09:40:16.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:16.955 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:16.958 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:16.990 INFO:tasks.rados.rados.0.smithi170.stdout:1290: done (1 left) 2024-03-27T09:40:16.990 INFO:tasks.rados.rados.0.smithi170.stdout:1292: done (0 left) 2024-03-27T09:40:16.990 INFO:tasks.rados.rados.0.smithi170.stdout:1293: delete oid 50 current snap is 149 2024-03-27T09:40:16.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:17.062 INFO:tasks.rados.rados.0.smithi170.stdout:1293: done (0 left) 2024-03-27T09:40:17.064 INFO:tasks.rados.rados.0.smithi170.stdout:1294: write (excl) oid 1 current snap is 149 2024-03-27T09:40:17.064 INFO:tasks.rados.rados.0.smithi170.stdout:1294: seq_num 622 ranges {453954=629471,1598388=515881} 2024-03-27T09:40:17.068 INFO:tasks.rados.rados.0.smithi170.stdout:1294: writing smithi17031214-1 from 453954 to 1083425 tid 1 2024-03-27T09:40:17.075 INFO:tasks.rados.rados.0.smithi170.stdout:1294: writing smithi17031214-1 from 1598388 to 2114269 tid 2 2024-03-27T09:40:17.075 INFO:tasks.rados.rados.0.smithi170.stdout:1295: snap_create 2024-03-27T09:40:17.108 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:17.105+0000 7f331ee13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:17.112 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:17.112+0000 7f331ee13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:17.134 INFO:tasks.rados.rados.0.smithi170.stdout:1294: finishing write tid 1 to smithi17031214-1 2024-03-27T09:40:17.137 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:17.169 INFO:tasks.rados.rados.0.smithi170.stdout:1294: finishing write tid 2 to smithi17031214-1 2024-03-27T09:40:17.177 INFO:tasks.rados.rados.0.smithi170.stdout:1294: finishing write tid 3 to smithi17031214-1 2024-03-27T09:40:17.180 INFO:tasks.rados.rados.0.smithi170.stdout:1294: finishing write tid 4 to smithi17031214-1 2024-03-27T09:40:17.180 INFO:tasks.rados.rados.0.smithi170.stdout:1294: oid 1 updating version 0 to 248 2024-03-27T09:40:17.180 INFO:tasks.rados.rados.0.smithi170.stdout:1294: oid 1 version 248 is already newer than 246 2024-03-27T09:40:17.181 INFO:tasks.rados.rados.0.smithi170.stdout:1294: oid 1 version 248 is already newer than 247 2024-03-27T09:40:17.181 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 248 (ObjNum 622 snap 149 seq_num 622) dirty exists 2024-03-27T09:40:17.181 INFO:tasks.rados.rados.0.smithi170.stdout:1294: left oid 1 (ObjNum 622 snap 149 seq_num 622) 2024-03-27T09:40:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:17.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:18.081 INFO:tasks.rados.rados.0.smithi170.stdout:1294: done (1 left) 2024-03-27T09:40:18.081 INFO:tasks.rados.rados.0.smithi170.stdout:1295: done (0 left) 2024-03-27T09:40:18.081 INFO:tasks.rados.rados.0.smithi170.stdout:1296: write (excl) oid 33 current snap is 150 2024-03-27T09:40:18.081 INFO:tasks.rados.rados.0.smithi170.stdout:1296: seq_num 623 ranges {470761=704720,1656870=659539} 2024-03-27T09:40:18.087 INFO:tasks.rados.rados.0.smithi170.stdout:1296: writing smithi17031214-33 from 470761 to 1175481 tid 1 2024-03-27T09:40:18.094 INFO:tasks.rados.rados.0.smithi170.stdout:1296: writing smithi17031214-33 from 1656870 to 2316409 tid 2 2024-03-27T09:40:18.094 INFO:tasks.rados.rados.0.smithi170.stdout:1297: snap_remove snap 148 2024-03-27T09:40:19.085 INFO:tasks.rados.rados.0.smithi170.stdout:1297: done (1 left) 2024-03-27T09:40:19.085 INFO:tasks.rados.rados.0.smithi170.stdout:1298: delete oid 41 current snap is 150 2024-03-27T09:40:19.085 INFO:tasks.rados.rados.0.smithi170.stdout:1296: finishing write tid 1 to smithi17031214-33 2024-03-27T09:40:19.085 INFO:tasks.rados.rados.0.smithi170.stdout:1296: finishing write tid 2 to smithi17031214-33 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: finishing write tid 3 to smithi17031214-33 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: finishing write tid 4 to smithi17031214-33 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: oid 33 updating version 0 to 335 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: oid 33 version 335 is already newer than 333 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: oid 33 version 335 is already newer than 334 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 335 (ObjNum 623 snap 150 seq_num 623) dirty exists 2024-03-27T09:40:19.086 INFO:tasks.rados.rados.0.smithi170.stdout:1296: left oid 33 (ObjNum 623 snap 150 seq_num 623) 2024-03-27T09:40:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:1296: done (1 left) 2024-03-27T09:40:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:1298: done (0 left) 2024-03-27T09:40:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:1299: snap_create 2024-03-27T09:40:19.200 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:19.312 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:19.309+0000 7fd5706ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:19.315 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:19.315+0000 7fd5706ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:19.339 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:19.385 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:20.088 INFO:tasks.rados.rados.0.smithi170.stdout:1299: done (0 left) 2024-03-27T09:40:20.089 INFO:tasks.rados.rados.0.smithi170.stdout:1300: delete oid 46 current snap is 151 2024-03-27T09:40:20.127 INFO:tasks.rados.rados.0.smithi170.stdout:1300: done (0 left) 2024-03-27T09:40:20.127 INFO:tasks.rados.rados.0.smithi170.stdout:1301: delete oid 16 current snap is 151 2024-03-27T09:40:20.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-03-27T09:40:20.202 INFO:tasks.rados.rados.0.smithi170.stdout:1301: done (0 left) 2024-03-27T09:40:20.202 INFO:tasks.rados.rados.0.smithi170.stdout:1302: write (excl) oid 24 current snap is 151 2024-03-27T09:40:20.203 INFO:tasks.rados.rados.0.smithi170.stdout:1302: seq_num 627 ranges {537989=722069,1890798=647191} 2024-03-27T09:40:20.208 INFO:tasks.rados.rados.0.smithi170.stdout:1302: writing smithi17031214-24 from 537989 to 1260058 tid 1 2024-03-27T09:40:20.215 INFO:tasks.rados.rados.0.smithi170.stdout:1302: writing smithi17031214-24 from 1890798 to 2537989 tid 2 2024-03-27T09:40:20.215 INFO:tasks.rados.rados.0.smithi170.stdout:1303: read oid 40 snap -1 2024-03-27T09:40:20.215 INFO:tasks.rados.rados.0.smithi170.stdout:1303: expect (ObjNum 601 snap 145 seq_num 601) 2024-03-27T09:40:20.215 INFO:tasks.rados.rados.0.smithi170.stdout:1304: write oid 30 current snap is 151 2024-03-27T09:40:20.215 INFO:tasks.rados.rados.0.smithi170.stdout:1304: seq_num 628 ranges {554796=797318,1949280=526888} 2024-03-27T09:40:20.223 INFO:tasks.rados.rados.0.smithi170.stdout:1304: writing smithi17031214-30 from 554796 to 1352114 tid 1 2024-03-27T09:40:20.230 INFO:tasks.rados.rados.0.smithi170.stdout:1304: writing smithi17031214-30 from 1949280 to 2476168 tid 2 2024-03-27T09:40:20.230 INFO:tasks.rados.rados.0.smithi170.stdout:1305: write (excl) oid 23 current snap is 151 2024-03-27T09:40:20.230 INFO:tasks.rados.rados.0.smithi170.stdout:1305: seq_num 629 ranges {571603=472567,1607762=670546} 2024-03-27T09:40:20.237 INFO:tasks.rados.rados.0.smithi170.stdout:1305: writing smithi17031214-23 from 571603 to 1044170 tid 1 2024-03-27T09:40:20.247 INFO:tasks.rados.rados.0.smithi170.stdout:1305: writing smithi17031214-23 from 1607762 to 2278308 tid 2 2024-03-27T09:40:20.247 INFO:tasks.rados.rados.0.smithi170.stdout:1306: snap_create 2024-03-27T09:40:20.248 INFO:tasks.rados.rados.0.smithi170.stdout:1303: expect (ObjNum 601 snap 145 seq_num 601) 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: finishing write tid 1 to smithi17031214-30 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1302: finishing write tid 1 to smithi17031214-24 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: finishing write tid 2 to smithi17031214-30 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1305: finishing write tid 1 to smithi17031214-23 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1302: finishing write tid 2 to smithi17031214-24 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: finishing write tid 3 to smithi17031214-30 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1302: finishing write tid 3 to smithi17031214-24 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: finishing write tid 4 to smithi17031214-30 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: oid 30 updating version 0 to 279 2024-03-27T09:40:20.401 INFO:tasks.rados.rados.0.smithi170.stdout:1304: oid 30 version 279 is already newer than 277 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1304: oid 30 version 279 is already newer than 279 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 279 (ObjNum 628 snap 151 seq_num 628) dirty exists 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1304: left oid 30 (ObjNum 628 snap 151 seq_num 628) 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1302: finishing write tid 4 to smithi17031214-24 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1302: oid 24 updating version 0 to 177 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1302: oid 24 updating version 177 to 178 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1302: oid 24 version 178 is already newer than 176 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 178 (ObjNum 627 snap 151 seq_num 627) dirty exists 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1302: left oid 24 (ObjNum 627 snap 151 seq_num 627) 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1305: finishing write tid 2 to smithi17031214-23 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1305: finishing write tid 3 to smithi17031214-23 2024-03-27T09:40:20.402 INFO:tasks.rados.rados.0.smithi170.stdout:1305: finishing write tid 4 to smithi17031214-23 2024-03-27T09:40:20.403 INFO:tasks.rados.rados.0.smithi170.stdout:1305: oid 23 updating version 0 to 528 2024-03-27T09:40:20.403 INFO:tasks.rados.rados.0.smithi170.stdout:1305: oid 23 updating version 528 to 530 2024-03-27T09:40:20.403 INFO:tasks.rados.rados.0.smithi170.stdout:1305: oid 23 version 530 is already newer than 529 2024-03-27T09:40:20.403 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 530 (ObjNum 629 snap 151 seq_num 629) dirty exists 2024-03-27T09:40:20.403 INFO:tasks.rados.rados.0.smithi170.stdout:1305: left oid 23 (ObjNum 629 snap 151 seq_num 629) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1302: done (4 left) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1303: done (3 left) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1304: done (2 left) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1305: done (1 left) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1306: done (0 left) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1307: read oid 1 snap 150 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1307: expect (ObjNum 622 snap 149 seq_num 622) 2024-03-27T09:40:21.092 INFO:tasks.rados.rados.0.smithi170.stdout:1308: snap_remove snap 147 2024-03-27T09:40:21.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:21.496 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:21.494+0000 7f6668b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:21.500 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:21.499+0000 7f6668b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:21.524 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:21.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:21.959 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:22.060 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:22.063 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:22.095 INFO:tasks.rados.rados.0.smithi170.stdout:1308: done (1 left) 2024-03-27T09:40:22.095 INFO:tasks.rados.rados.0.smithi170.stdout:1309: write oid 39 current snap is 152 2024-03-27T09:40:22.095 INFO:tasks.rados.rados.0.smithi170.stdout:1309: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-03-27T09:40:22.101 INFO:tasks.rados.rados.0.smithi170.stdout:1309: writing smithi17031214-39 from 588410 to 1136226 tid 1 2024-03-27T09:40:22.107 INFO:tasks.rados.rados.0.smithi170.stdout:1309: writing smithi17031214-39 from 1666244 to 2080448 tid 2 2024-03-27T09:40:22.110 INFO:tasks.rados.rados.0.smithi170.stdout:1309: writing smithi17031214-39 from 2484603 to 2588410 tid 3 2024-03-27T09:40:22.110 INFO:tasks.rados.rados.0.smithi170.stdout:1310: delete oid 34 current snap is 152 2024-03-27T09:40:22.110 INFO:tasks.rados.rados.0.smithi170.stdout:1307: expect (ObjNum 622 snap 149 seq_num 622) 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: finishing write tid 1 to smithi17031214-39 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: finishing write tid 2 to smithi17031214-39 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1307: done (2 left) 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1310: done (1 left) 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1311: write (excl) oid 10 current snap is 152 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: finishing write tid 3 to smithi17031214-39 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: finishing write tid 4 to smithi17031214-39 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: finishing write tid 5 to smithi17031214-39 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: oid 39 updating version 0 to 208 2024-03-27T09:40:22.260 INFO:tasks.rados.rados.0.smithi170.stdout:1309: oid 39 version 208 is already newer than 207 2024-03-27T09:40:22.261 INFO:tasks.rados.rados.0.smithi170.stdout:1309: oid 39 updating version 208 to 209 2024-03-27T09:40:22.261 INFO:tasks.rados.rados.0.smithi170.stdout:1309: oid 39 updating version 209 to 210 2024-03-27T09:40:22.261 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 210 (ObjNum 630 snap 152 seq_num 630) dirty exists 2024-03-27T09:40:22.261 INFO:tasks.rados.rados.0.smithi170.stdout:1309: left oid 39 (ObjNum 630 snap 152 seq_num 630) 2024-03-27T09:40:22.261 INFO:tasks.rados.rados.0.smithi170.stdout:1311: seq_num 632 ranges {622024=414667,1783208=417873} 2024-03-27T09:40:22.265 INFO:tasks.rados.rados.0.smithi170.stdout:1311: writing smithi17031214-10 from 622024 to 1036691 tid 1 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1311: writing smithi17031214-10 from 1783208 to 2201081 tid 2 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1309: done (1 left) 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1312: read oid 25 snap -1 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1312: expect deleted 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1313: write (excl) oid 2 current snap is 152 2024-03-27T09:40:22.272 INFO:tasks.rados.rados.0.smithi170.stdout:1313: seq_num 633 ranges {638831=489916,1841690=677884} 2024-03-27T09:40:22.278 INFO:tasks.rados.rados.0.smithi170.stdout:1313: writing smithi17031214-2 from 638831 to 1128747 tid 1 2024-03-27T09:40:22.286 INFO:tasks.rados.rados.0.smithi170.stdout:1313: writing smithi17031214-2 from 1841690 to 2519574 tid 2 2024-03-27T09:40:22.287 INFO:tasks.rados.rados.0.smithi170.stdout:1314: write oid 46 current snap is 152 2024-03-27T09:40:22.287 INFO:tasks.rados.rados.0.smithi170.stdout:1314: seq_num 634 ranges {655638=565165,1900172=421542} 2024-03-27T09:40:22.293 INFO:tasks.rados.rados.0.smithi170.stdout:1314: writing smithi17031214-46 from 655638 to 1220803 tid 1 2024-03-27T09:40:22.300 INFO:tasks.rados.rados.0.smithi170.stdout:1314: writing smithi17031214-46 from 1900172 to 2321714 tid 2 2024-03-27T09:40:22.300 INFO:tasks.rados.rados.0.smithi170.stdout:1315: read oid 19 snap -1 2024-03-27T09:40:22.300 INFO:tasks.rados.rados.0.smithi170.stdout:1315: expect deleted 2024-03-27T09:40:22.300 INFO:tasks.rados.rados.0.smithi170.stdout:1316: snap_remove snap 149 2024-03-27T09:40:23.098 INFO:tasks.rados.rados.0.smithi170.stdout:1316: done (5 left) 2024-03-27T09:40:23.098 INFO:tasks.rados.rados.0.smithi170.stdout:1317: snap_create 2024-03-27T09:40:23.098 INFO:tasks.rados.rados.0.smithi170.stdout:1314: finishing write tid 1 to smithi17031214-46 2024-03-27T09:40:23.098 INFO:tasks.rados.rados.0.smithi170.stdout:1314: finishing write tid 2 to smithi17031214-46 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1311: finishing write tid 1 to smithi17031214-10 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1313: finishing write tid 1 to smithi17031214-2 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1311: finishing write tid 2 to smithi17031214-10 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: finishing write tid 3 to smithi17031214-46 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: finishing write tid 4 to smithi17031214-46 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: oid 46 updating version 0 to 534 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: oid 46 updating version 534 to 536 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: oid 46 version 536 is already newer than 535 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 536 (ObjNum 634 snap 152 seq_num 634) dirty exists 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1314: left oid 46 (ObjNum 634 snap 152 seq_num 634) 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1313: finishing write tid 2 to smithi17031214-2 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1313: finishing write tid 3 to smithi17031214-2 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1311: finishing write tid 3 to smithi17031214-10 2024-03-27T09:40:23.099 INFO:tasks.rados.rados.0.smithi170.stdout:1313: finishing write tid 4 to smithi17031214-2 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1313: oid 2 updating version 0 to 463 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1313: oid 2 updating version 463 to 464 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1313: oid 2 updating version 464 to 465 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 465 (ObjNum 633 snap 152 seq_num 633) dirty exists 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1313: left oid 2 (ObjNum 633 snap 152 seq_num 633) 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1311: finishing write tid 4 to smithi17031214-10 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1311: oid 10 updating version 0 to 65 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1311: oid 10 version 65 is already newer than 64 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1311: oid 10 version 65 is already newer than 63 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 65 (ObjNum 632 snap 152 seq_num 632) dirty exists 2024-03-27T09:40:23.100 INFO:tasks.rados.rados.0.smithi170.stdout:1311: left oid 10 (ObjNum 632 snap 152 seq_num 632) 2024-03-27T09:40:23.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:23.684 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:23.681+0000 7f91cdd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:23.687 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:23.686+0000 7f91cdd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:23.711 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:23.758 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:23.759 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:24.102 INFO:tasks.rados.rados.0.smithi170.stdout:1311: done (5 left) 2024-03-27T09:40:24.102 INFO:tasks.rados.rados.0.smithi170.stdout:1312: done (4 left) 2024-03-27T09:40:24.102 INFO:tasks.rados.rados.0.smithi170.stdout:1313: done (3 left) 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1314: done (2 left) 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1315: done (1 left) 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1317: done (0 left) 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1318: read oid 22 snap -1 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1318: expect (ObjNum 588 snap 140 seq_num 588) 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1319: read oid 50 snap 150 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1319: expect deleted 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1320: write oid 30 current snap is 153 2024-03-27T09:40:24.103 INFO:tasks.rados.rados.0.smithi170.stdout:1320: seq_num 635 ranges {672445=640414,1842301=681553} 2024-03-27T09:40:24.110 INFO:tasks.rados.rados.0.smithi170.stdout:1320: writing smithi17031214-30 from 672445 to 1312859 tid 1 2024-03-27T09:40:24.120 INFO:tasks.rados.rados.0.smithi170.stdout:1320: writing smithi17031214-30 from 1842301 to 2523854 tid 2 2024-03-27T09:40:24.120 INFO:tasks.rados.rados.0.smithi170.stdout:1321: snap_remove snap 151 2024-03-27T09:40:25.106 INFO:tasks.rados.rados.0.smithi170.stdout:1321: done (3 left) 2024-03-27T09:40:25.106 INFO:tasks.rados.rados.0.smithi170.stdout:1322: snap_create 2024-03-27T09:40:25.106 INFO:tasks.rados.rados.0.smithi170.stdout:1318: expect (ObjNum 588 snap 140 seq_num 588) 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: finishing write tid 1 to smithi17031214-30 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: finishing write tid 2 to smithi17031214-30 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: finishing write tid 3 to smithi17031214-30 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: finishing write tid 4 to smithi17031214-30 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: oid 30 updating version 0 to 287 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: oid 30 version 287 is already newer than 285 2024-03-27T09:40:25.399 INFO:tasks.rados.rados.0.smithi170.stdout:1320: oid 30 version 287 is already newer than 286 2024-03-27T09:40:25.400 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 287 (ObjNum 635 snap 153 seq_num 635) dirty exists 2024-03-27T09:40:25.400 INFO:tasks.rados.rados.0.smithi170.stdout:1320: left oid 30 (ObjNum 635 snap 153 seq_num 635) 2024-03-27T09:40:25.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2024-03-27T09:40:25.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:25.871 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:25.868+0000 7f1833f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:25.875 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:25.873+0000 7f1833f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:25.898 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:25.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:25.991 INFO:tasks.rados.rados.0.smithi170.stdout:1318: done (3 left) 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1319: done (2 left) 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1320: done (1 left) 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1322: done (0 left) 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1323: read oid 30 snap -1 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1323: expect (ObjNum 635 snap 153 seq_num 635) 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1324: write (excl) oid 43 current snap is 154 2024-03-27T09:40:25.992 INFO:tasks.rados.rados.0.smithi170.stdout:1324: seq_num 636 ranges {689252=715663,1900783=425211} 2024-03-27T09:40:26.002 INFO:tasks.rados.rados.0.smithi170.stdout:1324: writing smithi17031214-43 from 689252 to 1404915 tid 1 2024-03-27T09:40:26.010 INFO:tasks.rados.rados.0.smithi170.stdout:1324: writing smithi17031214-43 from 1900783 to 2325994 tid 2 2024-03-27T09:40:26.010 INFO:tasks.rados.rados.0.smithi170.stdout:1325: snap_create 2024-03-27T09:40:26.051 INFO:tasks.rados.rados.0.smithi170.stdout:1323: expect (ObjNum 635 snap 153 seq_num 635) 2024-03-27T09:40:26.256 INFO:tasks.rados.rados.0.smithi170.stdout:1324: finishing write tid 1 to smithi17031214-43 2024-03-27T09:40:26.256 INFO:tasks.rados.rados.0.smithi170.stdout:1324: finishing write tid 2 to smithi17031214-43 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: finishing write tid 3 to smithi17031214-43 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: finishing write tid 4 to smithi17031214-43 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: oid 43 updating version 0 to 473 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: oid 43 version 473 is already newer than 472 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: oid 43 version 473 is already newer than 471 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 473 (ObjNum 636 snap 154 seq_num 636) dirty exists 2024-03-27T09:40:26.257 INFO:tasks.rados.rados.0.smithi170.stdout:1324: left oid 43 (ObjNum 636 snap 154 seq_num 636) 2024-03-27T09:40:26.993 INFO:tasks.rados.rados.0.smithi170.stdout:1323: done (2 left) 2024-03-27T09:40:26.993 INFO:tasks.rados.rados.0.smithi170.stdout:1324: done (1 left) 2024-03-27T09:40:26.993 INFO:tasks.rados.rados.0.smithi170.stdout:1325: done (0 left) 2024-03-27T09:40:26.993 INFO:tasks.rados.rados.0.smithi170.stdout:1326: write oid 26 current snap is 155 2024-03-27T09:40:26.993 INFO:tasks.rados.rados.0.smithi170.stdout:1326: seq_num 637 ranges {706059=790912,1959265=685222} 2024-03-27T09:40:27.001 INFO:tasks.rados.rados.0.smithi170.stdout:1326: writing smithi17031214-26 from 706059 to 1496971 tid 1 2024-03-27T09:40:27.007 INFO:tasks.rados.rados.0.smithi170.stdout:1326: writing smithi17031214-26 from 1959265 to 2644487 tid 2 2024-03-27T09:40:27.007 INFO:tasks.rados.rados.0.smithi170.stdout:1327: delete oid 15 current snap is 155 2024-03-27T09:40:27.034 INFO:tasks.rados.rados.0.smithi170.stdout:1327: done (1 left) 2024-03-27T09:40:27.034 INFO:tasks.rados.rados.0.smithi170.stdout:1328: delete oid 41 current snap is 155 2024-03-27T09:40:27.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1328: done (1 left) 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1329: read oid 10 snap -1 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1329: expect (ObjNum 632 snap 152 seq_num 632) 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1330: read oid 42 snap 153 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1330: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1331: read oid 13 snap 152 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1331: expect deleted 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1332: read oid 11 snap -1 2024-03-27T09:40:27.096 INFO:tasks.rados.rados.0.smithi170.stdout:1332: expect (ObjNum 559 snap 131 seq_num 559) 2024-03-27T09:40:27.097 INFO:tasks.rados.rados.0.smithi170.stdout:1333: delete oid 29 current snap is 155 2024-03-27T09:40:27.099 INFO:tasks.rados.rados.0.smithi170.stdout:1326: finishing write tid 1 to smithi17031214-26 2024-03-27T09:40:27.133 INFO:tasks.rados.rados.0.smithi170.stdout:1329: expect (ObjNum 632 snap 152 seq_num 632) 2024-03-27T09:40:27.188 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:27.191 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:27.300 INFO:tasks.rados.rados.0.smithi170.stdout:1326: finishing write tid 2 to smithi17031214-26 2024-03-27T09:40:27.300 INFO:tasks.rados.rados.0.smithi170.stdout:1330: expect (ObjNum 425 snap 100 seq_num 425) 2024-03-27T09:40:27.419 INFO:tasks.rados.rados.0.smithi170.stdout:1332: expect (ObjNum 559 snap 131 seq_num 559) 2024-03-27T09:40:27.544 INFO:tasks.rados.rados.0.smithi170.stdout:1326: finishing write tid 3 to smithi17031214-26 2024-03-27T09:40:27.544 INFO:tasks.rados.rados.0.smithi170.stdout:1326: finishing write tid 4 to smithi17031214-26 2024-03-27T09:40:27.544 INFO:tasks.rados.rados.0.smithi170.stdout:1326: oid 26 updating version 0 to 336 2024-03-27T09:40:27.544 INFO:tasks.rados.rados.0.smithi170.stdout:1326: oid 26 version 336 is already newer than 335 2024-03-27T09:40:27.544 INFO:tasks.rados.rados.0.smithi170.stdout:1326: oid 26 version 336 is already newer than 334 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 336 (ObjNum 637 snap 155 seq_num 637) dirty exists 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1326: left oid 26 (ObjNum 637 snap 155 seq_num 637) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1326: done (5 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1329: done (4 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1330: done (3 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1331: done (2 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1332: done (1 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1333: done (0 left) 2024-03-27T09:40:27.545 INFO:tasks.rados.rados.0.smithi170.stdout:1334: delete oid 49 current snap is 155 2024-03-27T09:40:27.552 INFO:tasks.rados.rados.0.smithi170.stdout:1334: done (0 left) 2024-03-27T09:40:27.552 INFO:tasks.rados.rados.0.smithi170.stdout:1335: read oid 48 snap -1 2024-03-27T09:40:27.552 INFO:tasks.rados.rados.0.smithi170.stdout:1335: expect (ObjNum 596 snap 142 seq_num 596) 2024-03-27T09:40:27.552 INFO:tasks.rados.rados.0.smithi170.stdout:1336: write (excl) oid 2 current snap is 155 2024-03-27T09:40:27.552 INFO:tasks.rados.rados.0.smithi170.stdout:1336: seq_num 642 ranges {790094=483510,1851675=436218} 2024-03-27T09:40:27.559 INFO:tasks.rados.rados.0.smithi170.stdout:1336: writing smithi17031214-2 from 790094 to 1273604 tid 1 2024-03-27T09:40:27.565 INFO:tasks.rados.rados.0.smithi170.stdout:1336: writing smithi17031214-2 from 1851675 to 2287893 tid 2 2024-03-27T09:40:27.565 INFO:tasks.rados.rados.0.smithi170.stdout:1337: snap_create 2024-03-27T09:40:27.573 INFO:tasks.rados.rados.0.smithi170.stdout:1335: expect (ObjNum 596 snap 142 seq_num 596) 2024-03-27T09:40:27.694 INFO:tasks.rados.rados.0.smithi170.stdout:1336: finishing write tid 1 to smithi17031214-2 2024-03-27T09:40:27.694 INFO:tasks.rados.rados.0.smithi170.stdout:1336: finishing write tid 2 to smithi17031214-2 2024-03-27T09:40:27.694 INFO:tasks.rados.rados.0.smithi170.stdout:1336: finishing write tid 3 to smithi17031214-2 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:1336: finishing write tid 4 to smithi17031214-2 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:1336: oid 2 updating version 0 to 475 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:1336: oid 2 updating version 475 to 476 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:1336: oid 2 updating version 476 to 477 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 477 (ObjNum 642 snap 155 seq_num 642) dirty exists 2024-03-27T09:40:27.695 INFO:tasks.rados.rados.0.smithi170.stdout:1336: left oid 2 (ObjNum 642 snap 155 seq_num 642) 2024-03-27T09:40:27.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:28.055 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:28.053+0000 7f7de91d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:28.058 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:28.058+0000 7f7de91d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:28.084 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:28.111 INFO:tasks.rados.rados.0.smithi170.stdout:1335: done (2 left) 2024-03-27T09:40:28.112 INFO:tasks.rados.rados.0.smithi170.stdout:1336: done (1 left) 2024-03-27T09:40:28.112 INFO:tasks.rados.rados.0.smithi170.stdout:1337: done (0 left) 2024-03-27T09:40:28.112 INFO:tasks.rados.rados.0.smithi170.stdout:1338: write oid 23 current snap is 156 2024-03-27T09:40:28.112 INFO:tasks.rados.rados.0.smithi170.stdout:1338: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-03-27T09:40:28.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: writing smithi17031214-23 from 406901 to 965660 tid 1 2024-03-27T09:40:28.122 INFO:tasks.rados.rados.0.smithi170.stdout:1338: writing smithi17031214-23 from 1393804 to 1973680 tid 2 2024-03-27T09:40:28.127 INFO:tasks.rados.rados.0.smithi170.stdout:1338: writing smithi17031214-23 from 2608396 to 2806901 tid 3 2024-03-27T09:40:28.127 INFO:tasks.rados.rados.0.smithi170.stdout:1339: snap_remove snap 154 2024-03-27T09:40:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:28.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:29.115 INFO:tasks.rados.rados.0.smithi170.stdout:1339: done (1 left) 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1340: delete oid 21 current snap is 156 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: finishing write tid 1 to smithi17031214-23 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: finishing write tid 2 to smithi17031214-23 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: finishing write tid 3 to smithi17031214-23 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: finishing write tid 4 to smithi17031214-23 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: finishing write tid 5 to smithi17031214-23 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: oid 23 updating version 0 to 540 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: oid 23 updating version 540 to 543 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: oid 23 version 543 is already newer than 543 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: oid 23 version 543 is already newer than 541 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 543 (ObjNum 643 snap 156 seq_num 643) dirty exists 2024-03-27T09:40:29.116 INFO:tasks.rados.rados.0.smithi170.stdout:1338: left oid 23 (ObjNum 643 snap 156 seq_num 643) 2024-03-27T09:40:29.193 INFO:tasks.rados.rados.0.smithi170.stdout:1338: done (1 left) 2024-03-27T09:40:29.194 INFO:tasks.rados.rados.0.smithi170.stdout:1340: done (0 left) 2024-03-27T09:40:29.194 INFO:tasks.rados.rados.0.smithi170.stdout:1341: write oid 46 current snap is 156 2024-03-27T09:40:29.194 INFO:tasks.rados.rados.0.smithi170.stdout:1341: seq_num 645 ranges {440515=709257,1910768=583545} 2024-03-27T09:40:29.202 INFO:tasks.rados.rados.0.smithi170.stdout:1341: writing smithi17031214-46 from 440515 to 1149772 tid 1 2024-03-27T09:40:29.213 INFO:tasks.rados.rados.0.smithi170.stdout:1341: writing smithi17031214-46 from 1910768 to 2494313 tid 2 2024-03-27T09:40:29.213 INFO:tasks.rados.rados.0.smithi170.stdout:1342: write oid 25 current snap is 156 2024-03-27T09:40:29.213 INFO:tasks.rados.rados.0.smithi170.stdout:1342: seq_num 646 ranges {457322=784506,1969250=443556} 2024-03-27T09:40:29.223 INFO:tasks.rados.rados.0.smithi170.stdout:1342: writing smithi17031214-25 from 457322 to 1241828 tid 1 2024-03-27T09:40:29.231 INFO:tasks.rados.rados.0.smithi170.stdout:1342: writing smithi17031214-25 from 1969250 to 2412806 tid 2 2024-03-27T09:40:29.231 INFO:tasks.rados.rados.0.smithi170.stdout:1343: write (excl) oid 15 current snap is 156 2024-03-27T09:40:29.231 INFO:tasks.rados.rados.0.smithi170.stdout:1343: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-03-27T09:40:29.239 INFO:tasks.rados.rados.0.smithi170.stdout:1343: writing smithi17031214-15 from 474129 to 1050237 tid 1 2024-03-27T09:40:29.246 INFO:tasks.rados.rados.0.smithi170.stdout:1343: writing smithi17031214-15 from 1627732 to 2214946 tid 2 2024-03-27T09:40:29.251 INFO:tasks.rados.rados.0.smithi170.stdout:1343: writing smithi17031214-15 from 2718088 to 2874129 tid 3 2024-03-27T09:40:29.251 INFO:tasks.rados.rados.0.smithi170.stdout:1344: snap_remove snap 150 2024-03-27T09:40:30.120 INFO:tasks.rados.rados.0.smithi170.stdout:1344: done (3 left) 2024-03-27T09:40:30.120 INFO:tasks.rados.rados.0.smithi170.stdout:1345: snap_remove snap 155 2024-03-27T09:40:30.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:30.278 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:30.276+0000 7f2cfa95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:30.282 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:30.281+0000 7f2cfa95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:30.306 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:30.353 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:31.122 INFO:tasks.rados.rados.0.smithi170.stdout:1345: done (3 left) 2024-03-27T09:40:31.123 INFO:tasks.rados.rados.0.smithi170.stdout:1346: snap_remove snap 152 2024-03-27T09:40:31.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1346: done (3 left) 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1347: delete oid 24 current snap is 156 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: finishing write tid 1 to smithi17031214-25 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: finishing write tid 2 to smithi17031214-25 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1343: finishing write tid 1 to smithi17031214-15 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: finishing write tid 3 to smithi17031214-25 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: finishing write tid 4 to smithi17031214-25 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: oid 25 updating version 0 to 480 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: oid 25 updating version 480 to 481 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: oid 25 updating version 481 to 482 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 482 (ObjNum 646 snap 156 seq_num 646) dirty exists 2024-03-27T09:40:32.127 INFO:tasks.rados.rados.0.smithi170.stdout:1342: left oid 25 (ObjNum 646 snap 156 seq_num 646) 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: finishing write tid 2 to smithi17031214-15 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1341: finishing write tid 1 to smithi17031214-46 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: finishing write tid 3 to smithi17031214-15 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1341: finishing write tid 2 to smithi17031214-46 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: finishing write tid 4 to smithi17031214-15 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1341: finishing write tid 3 to smithi17031214-46 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: finishing write tid 5 to smithi17031214-15 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: oid 15 updating version 0 to 184 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: oid 15 updating version 184 to 187 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: oid 15 version 187 is already newer than 187 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: oid 15 version 187 is already newer than 185 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 187 (ObjNum 647 snap 156 seq_num 647) dirty exists 2024-03-27T09:40:32.128 INFO:tasks.rados.rados.0.smithi170.stdout:1343: left oid 15 (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:1341: finishing write tid 4 to smithi17031214-46 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:1341: oid 46 updating version 0 to 547 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:1341: oid 46 updating version 547 to 549 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:1341: oid 46 version 549 is already newer than 549 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 549 (ObjNum 645 snap 156 seq_num 645) dirty exists 2024-03-27T09:40:32.129 INFO:tasks.rados.rados.0.smithi170.stdout:1341: left oid 46 (ObjNum 645 snap 156 seq_num 645) 2024-03-27T09:40:32.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1341: done (3 left) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1342: done (2 left) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1343: done (1 left) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1347: done (0 left) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1348: read oid 29 snap 153 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1348: expect (ObjNum 605 snap 145 seq_num 605) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1349: read oid 27 snap -1 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1349: expect (ObjNum 618 snap 146 seq_num 618) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1350: read oid 12 snap -1 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1350: expect (ObjNum 550 snap 131 seq_num 550) 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1351: read oid 8 snap -1 2024-03-27T09:40:32.210 INFO:tasks.rados.rados.0.smithi170.stdout:1351: expect (ObjNum 580 snap 138 seq_num 580) 2024-03-27T09:40:32.211 INFO:tasks.rados.rados.0.smithi170.stdout:1352: delete oid 23 current snap is 156 2024-03-27T09:40:32.270 INFO:tasks.rados.rados.0.smithi170.stdout:1352: done (4 left) 2024-03-27T09:40:32.270 INFO:tasks.rados.rados.0.smithi170.stdout:1353: write oid 44 current snap is 156 2024-03-27T09:40:32.270 INFO:tasks.rados.rados.0.smithi170.stdout:1353: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2024-03-27T09:40:32.275 INFO:tasks.rados.rados.0.smithi170.stdout:1353: writing smithi17031214-44 from 524550 to 926405 tid 1 2024-03-27T09:40:32.280 INFO:tasks.rados.rados.0.smithi170.stdout:1353: writing smithi17031214-44 from 1403178 to 1854072 tid 2 2024-03-27T09:40:32.286 INFO:tasks.rados.rados.0.smithi170.stdout:1353: writing smithi17031214-44 from 2516710 to 2924550 tid 3 2024-03-27T09:40:32.286 INFO:tasks.rados.rados.0.smithi170.stdout:1354: snap_create 2024-03-27T09:40:32.298 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:32.301 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:32.338 INFO:tasks.rados.rados.0.smithi170.stdout:1351: expect (ObjNum 580 snap 138 seq_num 580) 2024-03-27T09:40:32.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:32.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:32.464+0000 7f639e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:32.470 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:32.470+0000 7f639e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:32.496 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:32.541 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:32.545 INFO:tasks.rados.rados.0.smithi170.stdout:1349: expect (ObjNum 618 snap 146 seq_num 618) 2024-03-27T09:40:32.701 INFO:tasks.rados.rados.0.smithi170.stdout:1353: finishing write tid 1 to smithi17031214-44 2024-03-27T09:40:32.701 INFO:tasks.rados.rados.0.smithi170.stdout:1348: expect (ObjNum 605 snap 145 seq_num 605) 2024-03-27T09:40:32.781 INFO:tasks.rados.rados.0.smithi170.stdout:1353: finishing write tid 2 to smithi17031214-44 2024-03-27T09:40:32.781 INFO:tasks.rados.rados.0.smithi170.stdout:1350: expect (ObjNum 550 snap 131 seq_num 550) 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: finishing write tid 3 to smithi17031214-44 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: finishing write tid 4 to smithi17031214-44 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: finishing write tid 5 to smithi17031214-44 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: oid 44 updating version 0 to 410 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: oid 44 updating version 410 to 413 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: oid 44 version 413 is already newer than 412 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: oid 44 version 413 is already newer than 411 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 413 (ObjNum 650 snap 156 seq_num 650) dirty exists 2024-03-27T09:40:33.000 INFO:tasks.rados.rados.0.smithi170.stdout:1353: left oid 44 (ObjNum 650 snap 156 seq_num 650) 2024-03-27T09:40:33.129 INFO:tasks.rados.rados.0.smithi170.stdout:1348: done (5 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1349: done (4 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1350: done (3 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1351: done (2 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1353: done (1 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1354: done (0 left) 2024-03-27T09:40:33.130 INFO:tasks.rados.rados.0.smithi170.stdout:1355: delete oid 16 current snap is 157 2024-03-27T09:40:33.184 INFO:tasks.rados.rados.0.smithi170.stdout:1355: done (0 left) 2024-03-27T09:40:33.184 INFO:tasks.rados.rados.0.smithi170.stdout:1356: snap_create 2024-03-27T09:40:34.133 INFO:tasks.rados.rados.0.smithi170.stdout:1356: done (0 left) 2024-03-27T09:40:34.133 INFO:tasks.rados.rados.0.smithi170.stdout:1357: write (excl) oid 50 current snap is 158 2024-03-27T09:40:34.133 INFO:tasks.rados.rados.0.smithi170.stdout:1357: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-03-27T09:40:34.139 INFO:tasks.rados.rados.0.smithi170.stdout:1357: writing smithi17031214-50 from 558164 to 1110517 tid 1 2024-03-27T09:40:34.145 INFO:tasks.rados.rados.0.smithi170.stdout:1357: writing smithi17031214-50 from 1803789 to 2258352 tid 2 2024-03-27T09:40:34.149 INFO:tasks.rados.rados.0.smithi170.stdout:1357: writing smithi17031214-50 from 2855203 to 2958164 tid 3 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1358: read oid 37 snap 156 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1358: expect deleted 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1359: read oid 36 snap 153 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1359: expect deleted 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1360: read oid 2 snap 156 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1360: expect (ObjNum 642 snap 155 seq_num 642) 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1361: read oid 15 snap -1 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1361: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1362: read oid 9 snap -1 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1362: expect (ObjNum 599 snap 144 seq_num 599) 2024-03-27T09:40:34.150 INFO:tasks.rados.rados.0.smithi170.stdout:1363: snap_create 2024-03-27T09:40:34.218 INFO:tasks.rados.rados.0.smithi170.stdout:1361: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:34.339 INFO:tasks.rados.rados.0.smithi170.stdout:1362: expect (ObjNum 599 snap 144 seq_num 599) 2024-03-27T09:40:34.509 INFO:tasks.rados.rados.0.smithi170.stdout:1360: expect (ObjNum 642 snap 155 seq_num 642) 2024-03-27T09:40:34.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:34.649 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:34.646+0000 7f28f926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:34.652 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:34.651+0000 7f28f926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:34.689 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: finishing write tid 1 to smithi17031214-50 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: finishing write tid 2 to smithi17031214-50 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: finishing write tid 3 to smithi17031214-50 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: finishing write tid 4 to smithi17031214-50 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: finishing write tid 5 to smithi17031214-50 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: oid 50 updating version 0 to 416 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: oid 50 version 416 is already newer than 414 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: oid 50 version 416 is already newer than 415 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: oid 50 updating version 416 to 417 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 417 (ObjNum 652 snap 158 seq_num 652) dirty exists 2024-03-27T09:40:34.712 INFO:tasks.rados.rados.0.smithi170.stdout:1357: left oid 50 (ObjNum 652 snap 158 seq_num 652) 2024-03-27T09:40:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:34.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1357: done (6 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1358: done (5 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1359: done (4 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1360: done (3 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1361: done (2 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1362: done (1 left) 2024-03-27T09:40:35.137 INFO:tasks.rados.rados.0.smithi170.stdout:1363: done (0 left) 2024-03-27T09:40:35.138 INFO:tasks.rados.rados.0.smithi170.stdout:1364: snap_remove snap 158 2024-03-27T09:40:35.994 INFO:tasks.rados.rados.0.smithi170.stdout:1364: done (0 left) 2024-03-27T09:40:35.994 INFO:tasks.rados.rados.0.smithi170.stdout:1365: read oid 40 snap -1 2024-03-27T09:40:35.994 INFO:tasks.rados.rados.0.smithi170.stdout:1365: expect (ObjNum 601 snap 145 seq_num 601) 2024-03-27T09:40:35.994 INFO:tasks.rados.rados.0.smithi170.stdout:1366: snap_create 2024-03-27T09:40:36.056 INFO:tasks.rados.rados.0.smithi170.stdout:1365: expect (ObjNum 601 snap 145 seq_num 601) 2024-03-27T09:40:36.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-03-27T09:40:36.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:36.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:36.843+0000 7f982c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:36.849 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:36.848+0000 7f982c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:36.873 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:36.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:36.994 INFO:tasks.rados.rados.0.smithi170.stdout:1365: done (1 left) 2024-03-27T09:40:36.994 INFO:tasks.rados.rados.0.smithi170.stdout:1366: done (0 left) 2024-03-27T09:40:36.994 INFO:tasks.rados.rados.0.smithi170.stdout:1367: snap_remove snap 156 2024-03-27T09:40:37.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:37.404 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:37.407 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:37.995 INFO:tasks.rados.rados.0.smithi170.stdout:1367: done (0 left) 2024-03-27T09:40:37.995 INFO:tasks.rados.rados.0.smithi170.stdout:1368: snap_create 2024-03-27T09:40:38.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:38.996 INFO:tasks.rados.rados.0.smithi170.stdout:1368: done (0 left) 2024-03-27T09:40:38.996 INFO:tasks.rados.rados.0.smithi170.stdout:1369: write (excl) oid 17 current snap is 161 2024-03-27T09:40:38.996 INFO:tasks.rados.rados.0.smithi170.stdout:1369: seq_num 653 ranges {574971=627602,1862271=598221} 2024-03-27T09:40:39.002 INFO:tasks.rados.rados.0.smithi170.stdout:1369: writing smithi17031214-17 from 574971 to 1202573 tid 1 2024-03-27T09:40:39.010 INFO:tasks.rados.rados.0.smithi170.stdout:1369: writing smithi17031214-17 from 1862271 to 2460492 tid 2 2024-03-27T09:40:39.010 INFO:tasks.rados.rados.0.smithi170.stdout:1370: snap_create 2024-03-27T09:40:39.038 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:39.036+0000 7ff41c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:39.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:39.042+0000 7ff41c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:39.066 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:39.103 INFO:tasks.rados.rados.0.smithi170.stdout:1369: finishing write tid 1 to smithi17031214-17 2024-03-27T09:40:39.129 INFO:tasks.rados.rados.0.smithi170.stdout:1369: finishing write tid 2 to smithi17031214-17 2024-03-27T09:40:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:39.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:39.145 INFO:tasks.rados.rados.0.smithi170.stdout:1369: finishing write tid 3 to smithi17031214-17 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:1369: finishing write tid 4 to smithi17031214-17 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:1369: oid 17 updating version 0 to 420 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:1369: oid 17 version 420 is already newer than 419 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:1369: oid 17 updating version 420 to 421 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 421 (ObjNum 653 snap 161 seq_num 653) dirty exists 2024-03-27T09:40:39.147 INFO:tasks.rados.rados.0.smithi170.stdout:1369: left oid 17 (ObjNum 653 snap 161 seq_num 653) 2024-03-27T09:40:40.002 INFO:tasks.rados.rados.0.smithi170.stdout:1369: done (1 left) 2024-03-27T09:40:40.002 INFO:tasks.rados.rados.0.smithi170.stdout:1370: done (0 left) 2024-03-27T09:40:40.002 INFO:tasks.rados.rados.0.smithi170.stdout:1371: delete oid 13 current snap is 162 2024-03-27T09:40:40.057 INFO:tasks.rados.rados.0.smithi170.stdout:1371: done (0 left) 2024-03-27T09:40:40.057 INFO:tasks.rados.rados.0.smithi170.stdout:1372: snap_create 2024-03-27T09:40:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:1372: done (0 left) 2024-03-27T09:40:41.007 INFO:tasks.rados.rados.0.smithi170.stdout:1373: snap_remove snap 153 2024-03-27T09:40:41.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:41.246 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:41.244+0000 7f7f2406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:41.250 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:41.249+0000 7f7f2406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:41.274 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:41.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:42.010 INFO:tasks.rados.rados.0.smithi170.stdout:1373: done (0 left) 2024-03-27T09:40:42.010 INFO:tasks.rados.rados.0.smithi170.stdout:1374: delete oid 1 current snap is 163 2024-03-27T09:40:42.139 INFO:tasks.rados.rados.0.smithi170.stdout:1374: done (0 left) 2024-03-27T09:40:42.139 INFO:tasks.rados.rados.0.smithi170.stdout:1375: delete oid 38 current snap is 163 2024-03-27T09:40:42.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2024-03-27T09:40:42.174 INFO:tasks.rados.rados.0.smithi170.stdout:1375: done (0 left) 2024-03-27T09:40:42.175 INFO:tasks.rados.rados.0.smithi170.stdout:1376: write (excl) oid 33 current snap is 163 2024-03-27T09:40:42.175 INFO:tasks.rados.rados.0.smithi170.stdout:1376: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-03-27T09:40:42.181 INFO:tasks.rados.rados.0.smithi170.stdout:1376: writing smithi17031214-33 from 642199 to 1287150 tid 1 2024-03-27T09:40:42.187 INFO:tasks.rados.rados.0.smithi170.stdout:1376: writing smithi17031214-33 from 1696199 to 2301758 tid 2 2024-03-27T09:40:42.191 INFO:tasks.rados.rados.0.smithi170.stdout:1376: writing smithi17031214-33 from 2992318 to 3042199 tid 3 2024-03-27T09:40:42.192 INFO:tasks.rados.rados.0.smithi170.stdout:1377: snap_create 2024-03-27T09:40:42.277 INFO:tasks.rados.rados.0.smithi170.stdout:1376: finishing write tid 1 to smithi17031214-33 2024-03-27T09:40:42.295 INFO:tasks.rados.rados.0.smithi170.stdout:1376: finishing write tid 2 to smithi17031214-33 2024-03-27T09:40:42.303 INFO:tasks.rados.rados.0.smithi170.stdout:1376: finishing write tid 3 to smithi17031214-33 2024-03-27T09:40:42.308 INFO:tasks.rados.rados.0.smithi170.stdout:1376: finishing write tid 4 to smithi17031214-33 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: finishing write tid 5 to smithi17031214-33 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: oid 33 updating version 0 to 347 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: oid 33 version 347 is already newer than 344 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: oid 33 version 347 is already newer than 347 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: oid 33 version 347 is already newer than 345 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 347 (ObjNum 657 snap 163 seq_num 657) dirty exists 2024-03-27T09:40:42.310 INFO:tasks.rados.rados.0.smithi170.stdout:1376: left oid 33 (ObjNum 657 snap 163 seq_num 657) 2024-03-27T09:40:42.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:42.507 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:42.510 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:43.013 INFO:tasks.rados.rados.0.smithi170.stdout:1376: done (1 left) 2024-03-27T09:40:43.013 INFO:tasks.rados.rados.0.smithi170.stdout:1377: done (0 left) 2024-03-27T09:40:43.013 INFO:tasks.rados.rados.0.smithi170.stdout:1378: read oid 17 snap -1 2024-03-27T09:40:43.014 INFO:tasks.rados.rados.0.smithi170.stdout:1378: expect (ObjNum 653 snap 161 seq_num 653) 2024-03-27T09:40:43.014 INFO:tasks.rados.rados.0.smithi170.stdout:1379: delete oid 23 current snap is 164 2024-03-27T09:40:43.057 INFO:tasks.rados.rados.0.smithi170.stdout:1379: done (1 left) 2024-03-27T09:40:43.057 INFO:tasks.rados.rados.0.smithi170.stdout:1380: write oid 4 current snap is 164 2024-03-27T09:40:43.057 INFO:tasks.rados.rados.0.smithi170.stdout:1380: seq_num 659 ranges {675813=795449,2096810=609228} 2024-03-27T09:40:43.065 INFO:tasks.rados.rados.0.smithi170.stdout:1380: writing smithi17031214-4 from 675813 to 1471262 tid 1 2024-03-27T09:40:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:1380: writing smithi17031214-4 from 2096810 to 2706038 tid 2 2024-03-27T09:40:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:1381: delete oid 22 current snap is 164 2024-03-27T09:40:43.114 INFO:tasks.rados.rados.0.smithi170.stdout:1380: finishing write tid 1 to smithi17031214-4 2024-03-27T09:40:43.118 INFO:tasks.rados.rados.0.smithi170.stdout:1378: expect (ObjNum 653 snap 161 seq_num 653) 2024-03-27T09:40:43.281 INFO:tasks.rados.rados.0.smithi170.stdout:1380: finishing write tid 2 to smithi17031214-4 2024-03-27T09:40:43.281 INFO:tasks.rados.rados.0.smithi170.stdout:1380: finishing write tid 3 to smithi17031214-4 2024-03-27T09:40:43.281 INFO:tasks.rados.rados.0.smithi170.stdout:1380: finishing write tid 4 to smithi17031214-4 2024-03-27T09:40:43.281 INFO:tasks.rados.rados.0.smithi170.stdout:1380: oid 4 updating version 0 to 70 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1380: oid 4 version 70 is already newer than 68 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1380: oid 4 version 70 is already newer than 70 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 70 (ObjNum 659 snap 164 seq_num 659) dirty exists 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1380: left oid 4 (ObjNum 659 snap 164 seq_num 659) 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1378: done (2 left) 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1380: done (1 left) 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1381: done (0 left) 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1382: write oid 8 current snap is 164 2024-03-27T09:40:43.282 INFO:tasks.rados.rados.0.smithi170.stdout:1382: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-03-27T09:40:43.287 INFO:tasks.rados.rados.0.smithi170.stdout:1382: writing smithi17031214-8 from 709427 to 1255374 tid 1 2024-03-27T09:40:43.294 INFO:tasks.rados.rados.0.smithi170.stdout:1382: writing smithi17031214-8 from 1813774 to 2426671 tid 2 2024-03-27T09:40:43.298 INFO:tasks.rados.rados.0.smithi170.stdout:1382: writing smithi17031214-8 from 2985657 to 3109427 tid 3 2024-03-27T09:40:43.298 INFO:tasks.rados.rados.0.smithi170.stdout:1383: snap_remove snap 159 2024-03-27T09:40:43.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:43.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:43.429+0000 7f9352f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:43.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:43.434+0000 7f9352f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:43.459 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:43.505 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1383: done (1 left) 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1384: read oid 15 snap -1 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1384: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1385: read oid 3 snap -1 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1385: expect deleted 2024-03-27T09:40:44.018 INFO:tasks.rados.rados.0.smithi170.stdout:1386: snap_remove snap 157 2024-03-27T09:40:45.019 INFO:tasks.rados.rados.0.smithi170.stdout:1386: done (3 left) 2024-03-27T09:40:45.019 INFO:tasks.rados.rados.0.smithi170.stdout:1387: snap_create 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: finishing write tid 1 to smithi17031214-8 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: finishing write tid 2 to smithi17031214-8 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: finishing write tid 3 to smithi17031214-8 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: finishing write tid 4 to smithi17031214-8 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: finishing write tid 5 to smithi17031214-8 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: oid 8 updating version 0 to 429 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: oid 8 version 429 is already newer than 428 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: oid 8 version 429 is already newer than 427 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: oid 8 updating version 429 to 430 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 430 (ObjNum 661 snap 164 seq_num 661) dirty exists 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1382: left oid 8 (ObjNum 661 snap 164 seq_num 661) 2024-03-27T09:40:45.020 INFO:tasks.rados.rados.0.smithi170.stdout:1384: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:45.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:45.620 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:45.618+0000 7fa47775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:45.623 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:45.623+0000 7fa47775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:45.648 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:45.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1382: done (3 left) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1384: done (2 left) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1385: done (1 left) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1387: done (0 left) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1388: read oid 43 snap -1 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1388: expect (ObjNum 636 snap 154 seq_num 636) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1389: read oid 5 snap -1 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1389: expect (ObjNum 548 snap 129 seq_num 548) 2024-03-27T09:40:45.996 INFO:tasks.rados.rados.0.smithi170.stdout:1390: read oid 48 snap -1 2024-03-27T09:40:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:1390: expect (ObjNum 596 snap 142 seq_num 596) 2024-03-27T09:40:45.997 INFO:tasks.rados.rados.0.smithi170.stdout:1391: delete oid 16 current snap is 165 2024-03-27T09:40:46.071 INFO:tasks.rados.rados.0.smithi170.stdout:1390: expect (ObjNum 596 snap 142 seq_num 596) 2024-03-27T09:40:46.192 INFO:tasks.rados.rados.0.smithi170.stdout:1389: expect (ObjNum 548 snap 129 seq_num 548) 2024-03-27T09:40:46.310 INFO:tasks.rados.rados.0.smithi170.stdout:1388: expect (ObjNum 636 snap 154 seq_num 636) 2024-03-27T09:40:46.465 INFO:tasks.rados.rados.0.smithi170.stdout:1388: done (3 left) 2024-03-27T09:40:46.465 INFO:tasks.rados.rados.0.smithi170.stdout:1389: done (2 left) 2024-03-27T09:40:46.465 INFO:tasks.rados.rados.0.smithi170.stdout:1390: done (1 left) 2024-03-27T09:40:46.465 INFO:tasks.rados.rados.0.smithi170.stdout:1391: done (0 left) 2024-03-27T09:40:46.465 INFO:tasks.rados.rados.0.smithi170.stdout:1392: snap_remove snap 163 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1392: done (0 left) 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1393: read oid 16 snap -1 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1393: expect deleted 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1394: read oid 3 snap -1 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1394: expect deleted 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1395: write (excl) oid 46 current snap is 165 2024-03-27T09:40:47.029 INFO:tasks.rados.rados.0.smithi170.stdout:1395: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-03-27T09:40:47.037 INFO:tasks.rados.rados.0.smithi170.stdout:1395: writing smithi17031214-46 from 743041 to 1155839 tid 1 2024-03-27T09:40:47.047 INFO:tasks.rados.rados.0.smithi170.stdout:1395: writing smithi17031214-46 from 1930738 to 2547304 tid 2 2024-03-27T09:40:47.051 INFO:tasks.rados.rados.0.smithi170.stdout:1395: writing smithi17031214-46 from 3040503 to 3143041 tid 3 2024-03-27T09:40:47.051 INFO:tasks.rados.rados.0.smithi170.stdout:1396: write (excl) oid 41 current snap is 165 2024-03-27T09:40:47.051 INFO:tasks.rados.rados.0.smithi170.stdout:1396: seq_num 664 ranges {759848=488047,1989220=760224} 2024-03-27T09:40:47.060 INFO:tasks.rados.rados.0.smithi170.stdout:1396: writing smithi17031214-41 from 759848 to 1247895 tid 1 2024-03-27T09:40:47.069 INFO:tasks.rados.rados.0.smithi170.stdout:1396: writing smithi17031214-41 from 1989220 to 2749444 tid 2 2024-03-27T09:40:47.069 INFO:tasks.rados.rados.0.smithi170.stdout:1397: read oid 35 snap 160 2024-03-27T09:40:47.069 INFO:tasks.rados.rados.0.smithi170.stdout:1397: expect (ObjNum 553 snap 131 seq_num 553) 2024-03-27T09:40:47.069 INFO:tasks.rados.rados.0.smithi170.stdout:1398: read oid 34 snap 164 2024-03-27T09:40:47.070 INFO:tasks.rados.rados.0.smithi170.stdout:1398: expect deleted 2024-03-27T09:40:47.070 INFO:tasks.rados.rados.0.smithi170.stdout:1399: write oid 9 current snap is 165 2024-03-27T09:40:47.070 INFO:tasks.rados.rados.0.smithi170.stdout:1399: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-03-27T09:40:47.078 INFO:tasks.rados.rados.0.smithi170.stdout:1399: writing smithi17031214-9 from 776655 to 1339951 tid 1 2024-03-27T09:40:47.087 INFO:tasks.rados.rados.0.smithi170.stdout:1399: writing smithi17031214-9 from 2047702 to 2667937 tid 2 2024-03-27T09:40:47.091 INFO:tasks.rados.rados.0.smithi170.stdout:1399: writing smithi17031214-9 from 3095349 to 3176655 tid 3 2024-03-27T09:40:47.091 INFO:tasks.rados.rados.0.smithi170.stdout:1400: read oid 15 snap -1 2024-03-27T09:40:47.091 INFO:tasks.rados.rados.0.smithi170.stdout:1400: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:47.091 INFO:tasks.rados.rados.0.smithi170.stdout:1401: write (excl) oid 14 current snap is 165 2024-03-27T09:40:47.091 INFO:tasks.rados.rados.0.smithi170.stdout:1401: seq_num 666 ranges {793462=638545,2106184=763893} 2024-03-27T09:40:47.101 INFO:tasks.rados.rados.0.smithi170.stdout:1401: writing smithi17031214-14 from 793462 to 1432007 tid 1 2024-03-27T09:40:47.113 INFO:tasks.rados.rados.0.smithi170.stdout:1401: writing smithi17031214-14 from 2106184 to 2870077 tid 2 2024-03-27T09:40:47.114 INFO:tasks.rados.rados.0.smithi170.stdout:1402: write oid 24 current snap is 165 2024-03-27T09:40:47.114 INFO:tasks.rados.rados.0.smithi170.stdout:1402: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-03-27T09:40:47.123 INFO:tasks.rados.rados.0.smithi170.stdout:1402: writing smithi17031214-24 from 410269 to 1124063 tid 1 2024-03-27T09:40:47.131 INFO:tasks.rados.rados.0.smithi170.stdout:1402: writing smithi17031214-24 from 1648313 to 2155864 tid 2 2024-03-27T09:40:47.137 INFO:tasks.rados.rados.0.smithi170.stdout:1402: writing smithi17031214-24 from 2917489 to 3210269 tid 3 2024-03-27T09:40:47.138 INFO:tasks.rados.rados.0.smithi170.stdout:1403: write (excl) oid 44 current snap is 165 2024-03-27T09:40:47.138 INFO:tasks.rados.rados.0.smithi170.stdout:1403: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-03-27T09:40:47.144 INFO:tasks.rados.rados.0.smithi170.stdout:1403: writing smithi17031214-44 from 427076 to 1216119 tid 1 2024-03-27T09:40:47.152 INFO:tasks.rados.rados.0.smithi170.stdout:1403: writing smithi17031214-44 from 1706795 to 2474357 tid 2 2024-03-27T09:40:47.158 INFO:tasks.rados.rados.0.smithi170.stdout:1403: writing smithi17031214-44 from 2944912 to 3227076 tid 3 2024-03-27T09:40:47.158 INFO:tasks.rados.rados.0.smithi170.stdout:1404: write oid 1 current snap is 165 2024-03-27T09:40:47.158 INFO:tasks.rados.rados.0.smithi170.stdout:1404: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-03-27T09:40:47.165 INFO:tasks.rados.rados.0.smithi170.stdout:1404: writing smithi17031214-1 from 443883 to 908175 tid 1 2024-03-27T09:40:47.173 INFO:tasks.rados.rados.0.smithi170.stdout:1404: writing smithi17031214-1 from 1365277 to 1876497 tid 2 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1404: writing smithi17031214-1 from 2572335 to 3243883 tid 3 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1393: done (11 left) 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1394: done (10 left) 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1405: read oid 38 snap -1 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1405: expect deleted 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1406: write (excl) oid 19 current snap is 165 2024-03-27T09:40:47.184 INFO:tasks.rados.rados.0.smithi170.stdout:1406: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-03-27T09:40:47.193 INFO:tasks.rados.rados.0.smithi170.stdout:1406: writing smithi17031214-19 from 460690 to 1116584 tid 1 2024-03-27T09:40:47.203 INFO:tasks.rados.rados.0.smithi170.stdout:1406: writing smithi17031214-19 from 1540112 to 2311343 tid 2 2024-03-27T09:40:47.212 INFO:tasks.rados.rados.0.smithi170.stdout:1406: writing smithi17031214-19 from 2716111 to 3255889 tid 3 2024-03-27T09:40:47.212 INFO:tasks.rados.rados.0.smithi170.stdout:1407: write oid 2 current snap is 165 2024-03-27T09:40:47.212 INFO:tasks.rados.rados.0.smithi170.stdout:1401: finishing write tid 1 to smithi17031214-14 2024-03-27T09:40:47.212 INFO:tasks.rados.rados.0.smithi170.stdout:1395: finishing write tid 1 to smithi17031214-46 2024-03-27T09:40:47.212 INFO:tasks.rados.rados.0.smithi170.stdout:1407: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-03-27T09:40:47.221 INFO:tasks.rados.rados.0.smithi170.stdout:1407: writing smithi17031214-2 from 477497 to 1208640 tid 1 2024-03-27T09:40:47.231 INFO:tasks.rados.rados.0.smithi170.stdout:1407: writing smithi17031214-2 from 1998594 to 2513483 tid 2 2024-03-27T09:40:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:1407: writing smithi17031214-2 from 3143534 to 3277497 tid 3 2024-03-27T09:40:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:1408: snap_remove snap 161 2024-03-27T09:40:47.236 INFO:tasks.rados.rados.0.smithi170.stdout:1396: finishing write tid 1 to smithi17031214-41 2024-03-27T09:40:47.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:47.611 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:47.615 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:47.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2024-03-27T09:40:47.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:47.805 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:47.802+0000 7f709219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:47.808 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:47.808+0000 7f709219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:47.832 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:47.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:47.878 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:48.032 INFO:tasks.rados.rados.0.smithi170.stdout:1408: done (13 left) 2024-03-27T09:40:48.032 INFO:tasks.rados.rados.0.smithi170.stdout:1409: snap_create 2024-03-27T09:40:48.032 INFO:tasks.rados.rados.0.smithi170.stdout:1400: expect (ObjNum 647 snap 156 seq_num 647) 2024-03-27T09:40:48.139 INFO:tasks.rados.rados.0.smithi170.stdout:1401: finishing write tid 2 to smithi17031214-14 2024-03-27T09:40:48.139 INFO:tasks.rados.rados.0.smithi170.stdout:1396: finishing write tid 2 to smithi17031214-41 2024-03-27T09:40:48.139 INFO:tasks.rados.rados.0.smithi170.stdout:1395: finishing write tid 2 to smithi17031214-46 2024-03-27T09:40:48.139 INFO:tasks.rados.rados.0.smithi170.stdout:1396: finishing write tid 3 to smithi17031214-41 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1403: finishing write tid 1 to smithi17031214-44 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1396: finishing write tid 4 to smithi17031214-41 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1396: oid 41 updating version 0 to 432 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1396: oid 41 updating version 432 to 434 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1396: oid 41 version 434 is already newer than 433 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 434 (ObjNum 664 snap 165 seq_num 664) dirty exists 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1396: left oid 41 (ObjNum 664 snap 165 seq_num 664) 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1402: finishing write tid 1 to smithi17031214-24 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1404: finishing write tid 1 to smithi17031214-1 2024-03-27T09:40:48.140 INFO:tasks.rados.rados.0.smithi170.stdout:1397: expect (ObjNum 553 snap 131 seq_num 553) 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: finishing write tid 3 to smithi17031214-14 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: finishing write tid 4 to smithi17031214-14 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: oid 14 updating version 0 to 297 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: oid 14 updating version 297 to 298 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: oid 14 version 298 is already newer than 296 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 298 (ObjNum 666 snap 165 seq_num 666) dirty exists 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1401: left oid 14 (ObjNum 666 snap 165 seq_num 666) 2024-03-27T09:40:48.260 INFO:tasks.rados.rados.0.smithi170.stdout:1395: finishing write tid 3 to smithi17031214-46 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: finishing write tid 4 to smithi17031214-46 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1402: finishing write tid 2 to smithi17031214-24 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: finishing write tid 5 to smithi17031214-46 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: oid 46 updating version 0 to 564 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: oid 46 version 564 is already newer than 563 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: oid 46 updating version 564 to 566 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: oid 46 version 566 is already newer than 566 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 566 (ObjNum 663 snap 165 seq_num 663) dirty exists 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1395: left oid 46 (ObjNum 663 snap 165 seq_num 663) 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1403: finishing write tid 2 to smithi17031214-44 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1402: finishing write tid 3 to smithi17031214-24 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1404: finishing write tid 2 to smithi17031214-1 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1406: finishing write tid 1 to smithi17031214-19 2024-03-27T09:40:48.261 INFO:tasks.rados.rados.0.smithi170.stdout:1403: finishing write tid 3 to smithi17031214-44 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: finishing write tid 4 to smithi17031214-24 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: finishing write tid 5 to smithi17031214-24 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: oid 24 updating version 0 to 197 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: oid 24 version 197 is already newer than 196 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: oid 24 version 197 is already newer than 194 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: oid 24 version 197 is already newer than 196 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 197 (ObjNum 667 snap 165 seq_num 667) dirty exists 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1402: left oid 24 (ObjNum 667 snap 165 seq_num 667) 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1407: finishing write tid 1 to smithi17031214-2 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1406: finishing write tid 2 to smithi17031214-19 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1403: finishing write tid 4 to smithi17031214-44 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1404: finishing write tid 3 to smithi17031214-1 2024-03-27T09:40:48.262 INFO:tasks.rados.rados.0.smithi170.stdout:1403: finishing write tid 5 to smithi17031214-44 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1403: oid 44 updating version 0 to 440 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1403: oid 44 version 440 is already newer than 438 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1403: oid 44 updating version 440 to 441 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1403: oid 44 version 441 is already newer than 440 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 441 (ObjNum 668 snap 165 seq_num 668) dirty exists 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1403: left oid 44 (ObjNum 668 snap 165 seq_num 668) 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1406: finishing write tid 3 to smithi17031214-19 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1407: finishing write tid 2 to smithi17031214-2 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1407: finishing write tid 3 to smithi17031214-2 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1399: finishing write tid 1 to smithi17031214-9 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1404: finishing write tid 4 to smithi17031214-1 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1406: finishing write tid 4 to smithi17031214-19 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1404: finishing write tid 5 to smithi17031214-1 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1404: oid 1 updating version 0 to 269 2024-03-27T09:40:48.263 INFO:tasks.rados.rados.0.smithi170.stdout:1404: oid 1 version 269 is already newer than 268 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1404: oid 1 version 269 is already newer than 267 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1404: oid 1 updating version 269 to 270 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 270 (ObjNum 669 snap 165 seq_num 669) dirty exists 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1404: left oid 1 (ObjNum 669 snap 165 seq_num 669) 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: finishing write tid 5 to smithi17031214-19 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: oid 19 updating version 0 to 353 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: oid 19 version 353 is already newer than 352 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: oid 19 version 353 is already newer than 351 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: oid 19 version 353 is already newer than 350 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 353 (ObjNum 670 snap 165 seq_num 670) dirty exists 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1406: left oid 19 (ObjNum 670 snap 165 seq_num 670) 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1407: finishing write tid 4 to smithi17031214-2 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1399: finishing write tid 2 to smithi17031214-9 2024-03-27T09:40:48.264 INFO:tasks.rados.rados.0.smithi170.stdout:1407: finishing write tid 5 to smithi17031214-2 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1407: oid 2 updating version 0 to 499 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1407: oid 2 version 499 is already newer than 496 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1407: oid 2 version 499 is already newer than 498 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1407: oid 2 version 499 is already newer than 498 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 499 (ObjNum 671 snap 165 seq_num 671) dirty exists 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1407: left oid 2 (ObjNum 671 snap 165 seq_num 671) 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: finishing write tid 3 to smithi17031214-9 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: finishing write tid 4 to smithi17031214-9 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: finishing write tid 5 to smithi17031214-9 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: oid 9 updating version 0 to 570 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: oid 9 version 570 is already newer than 568 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: oid 9 version 570 is already newer than 570 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: oid 9 updating version 570 to 571 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 571 (ObjNum 665 snap 165 seq_num 665) dirty exists 2024-03-27T09:40:48.265 INFO:tasks.rados.rados.0.smithi170.stdout:1399: left oid 9 (ObjNum 665 snap 165 seq_num 665) 2024-03-27T09:40:49.038 INFO:tasks.rados.rados.0.smithi170.stdout:1395: done (13 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1396: done (12 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1397: done (11 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1398: done (10 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1399: done (9 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1400: done (8 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1401: done (7 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1402: done (6 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1403: done (5 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1404: done (4 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1405: done (3 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1406: done (2 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1407: done (1 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1409: done (0 left) 2024-03-27T09:40:49.039 INFO:tasks.rados.rados.0.smithi170.stdout:1410: snap_create 2024-03-27T09:40:49.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:49.997 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:49.994+0000 7f9e83b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:50.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:50.000+0000 7f9e83b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:50.024 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:50.041 INFO:tasks.rados.rados.0.smithi170.stdout:1410: done (0 left) 2024-03-27T09:40:50.041 INFO:tasks.rados.rados.0.smithi170.stdout:1411: delete oid 25 current snap is 167 2024-03-27T09:40:50.106 INFO:tasks.rados.rados.0.smithi170.stdout:1411: done (0 left) 2024-03-27T09:40:50.106 INFO:tasks.rados.rados.0.smithi170.stdout:1412: write oid 35 current snap is 167 2024-03-27T09:40:50.107 INFO:tasks.rados.rados.0.smithi170.stdout:1412: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-03-27T09:40:50.111 INFO:tasks.rados.rados.0.smithi170.stdout:1412: writing smithi17031214-35 from 511111 to 992752 tid 1 2024-03-27T09:40:50.118 INFO:tasks.rados.rados.0.smithi170.stdout:1412: writing smithi17031214-35 from 1599205 to 2117763 tid 2 2024-03-27T09:40:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:50.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:50.127 INFO:tasks.rados.rados.0.smithi170.stdout:1412: writing smithi17031214-35 from 2682027 to 3171974 tid 3 2024-03-27T09:40:50.127 INFO:tasks.rados.rados.0.smithi170.stdout:1413: read oid 12 snap -1 2024-03-27T09:40:50.127 INFO:tasks.rados.rados.0.smithi170.stdout:1413: expect (ObjNum 550 snap 131 seq_num 550) 2024-03-27T09:40:50.127 INFO:tasks.rados.rados.0.smithi170.stdout:1414: write (excl) oid 10 current snap is 167 2024-03-27T09:40:50.127 INFO:tasks.rados.rados.0.smithi170.stdout:1414: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-03-27T09:40:50.135 INFO:tasks.rados.rados.0.smithi170.stdout:1414: writing smithi17031214-10 from 527918 to 1084808 tid 1 2024-03-27T09:40:50.144 INFO:tasks.rados.rados.0.smithi170.stdout:1414: writing smithi17031214-10 from 1657687 to 2436256 tid 2 2024-03-27T09:40:50.150 INFO:tasks.rados.rados.0.smithi170.stdout:1414: writing smithi17031214-10 from 3225803 to 3327918 tid 3 2024-03-27T09:40:50.151 INFO:tasks.rados.rados.0.smithi170.stdout:1415: write oid 2 current snap is 167 2024-03-27T09:40:50.151 INFO:tasks.rados.rados.0.smithi170.stdout:1415: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-03-27T09:40:50.160 INFO:tasks.rados.rados.0.smithi170.stdout:1415: writing smithi17031214-2 from 544725 to 1176864 tid 1 2024-03-27T09:40:50.168 INFO:tasks.rados.rados.0.smithi170.stdout:1415: writing smithi17031214-2 from 1716169 to 2238396 tid 2 2024-03-27T09:40:50.177 INFO:tasks.rados.rados.0.smithi170.stdout:1415: writing smithi17031214-2 from 2736873 to 3249364 tid 3 2024-03-27T09:40:50.177 INFO:tasks.rados.rados.0.smithi170.stdout:1416: write oid 40 current snap is 167 2024-03-27T09:40:50.177 INFO:tasks.rados.rados.0.smithi170.stdout:1416: seq_num 676 ranges {561532=707388,2058298=782238} 2024-03-27T09:40:50.187 INFO:tasks.rados.rados.0.smithi170.stdout:1416: writing smithi17031214-40 from 561532 to 1268920 tid 1 2024-03-27T09:40:50.200 INFO:tasks.rados.rados.0.smithi170.stdout:1416: writing smithi17031214-40 from 2058298 to 2840536 tid 2 2024-03-27T09:40:50.200 INFO:tasks.rados.rados.0.smithi170.stdout:1412: finishing write tid 1 to smithi17031214-35 2024-03-27T09:40:50.200 INFO:tasks.rados.rados.0.smithi170.stdout:1417: read oid 1 snap -1 2024-03-27T09:40:50.200 INFO:tasks.rados.rados.0.smithi170.stdout:1417: expect (ObjNum 669 snap 165 seq_num 669) 2024-03-27T09:40:50.200 INFO:tasks.rados.rados.0.smithi170.stdout:1418: delete oid 26 current snap is 167 2024-03-27T09:40:50.201 INFO:tasks.rados.rados.0.smithi170.stdout:1412: finishing write tid 2 to smithi17031214-35 2024-03-27T09:40:50.201 INFO:tasks.rados.rados.0.smithi170.stdout:1413: expect (ObjNum 550 snap 131 seq_num 550) 2024-03-27T09:40:50.426 INFO:tasks.rados.rados.0.smithi170.stdout:1414: finishing write tid 1 to smithi17031214-10 2024-03-27T09:40:50.426 INFO:tasks.rados.rados.0.smithi170.stdout:1412: finishing write tid 3 to smithi17031214-35 2024-03-27T09:40:50.426 INFO:tasks.rados.rados.0.smithi170.stdout:1412: finishing write tid 4 to smithi17031214-35 2024-03-27T09:40:50.426 INFO:tasks.rados.rados.0.smithi170.stdout:1412: finishing write tid 5 to smithi17031214-35 2024-03-27T09:40:50.426 INFO:tasks.rados.rados.0.smithi170.stdout:1412: oid 35 updating version 0 to 436 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1412: oid 35 updating version 436 to 438 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1412: oid 35 version 438 is already newer than 437 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1412: oid 35 updating version 438 to 439 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 439 (ObjNum 673 snap 167 seq_num 673) dirty exists 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1412: left oid 35 (ObjNum 673 snap 167 seq_num 673) 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1415: finishing write tid 1 to smithi17031214-2 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1412: done (6 left) 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1413: done (5 left) 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1418: done (4 left) 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1419: write oid 6 current snap is 167 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1414: finishing write tid 2 to smithi17031214-10 2024-03-27T09:40:50.427 INFO:tasks.rados.rados.0.smithi170.stdout:1419: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-03-27T09:40:50.432 INFO:tasks.rados.rados.0.smithi170.stdout:1419: writing smithi17031214-6 from 595146 to 1169385 tid 1 2024-03-27T09:40:50.439 INFO:tasks.rados.rados.0.smithi170.stdout:1419: writing smithi17031214-6 from 1891615 to 2561169 tid 2 2024-03-27T09:40:50.444 INFO:tasks.rados.rados.0.smithi170.stdout:1419: writing smithi17031214-6 from 3219142 to 3395146 tid 3 2024-03-27T09:40:50.444 INFO:tasks.rados.rados.0.smithi170.stdout:1420: read oid 23 snap -1 2024-03-27T09:40:50.444 INFO:tasks.rados.rados.0.smithi170.stdout:1420: expect deleted 2024-03-27T09:40:50.444 INFO:tasks.rados.rados.0.smithi170.stdout:1421: snap_remove snap 166 2024-03-27T09:40:51.043 INFO:tasks.rados.rados.0.smithi170.stdout:1421: done (6 left) 2024-03-27T09:40:51.044 INFO:tasks.rados.rados.0.smithi170.stdout:1422: write (excl) oid 50 current snap is 167 2024-03-27T09:40:51.044 INFO:tasks.rados.rados.0.smithi170.stdout:1414: finishing write tid 3 to smithi17031214-10 2024-03-27T09:40:51.044 INFO:tasks.rados.rados.0.smithi170.stdout:1422: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-03-27T09:40:51.050 INFO:tasks.rados.rados.0.smithi170.stdout:1422: writing smithi17031214-50 from 611953 to 1261441 tid 1 2024-03-27T09:40:51.056 INFO:tasks.rados.rados.0.smithi170.stdout:1422: writing smithi17031214-50 from 1950097 to 2479662 tid 2 2024-03-27T09:40:51.061 INFO:tasks.rados.rados.0.smithi170.stdout:1422: writing smithi17031214-50 from 3246565 to 3411953 tid 3 2024-03-27T09:40:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:1423: delete oid 26 current snap is 167 2024-03-27T09:40:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:1416: finishing write tid 1 to smithi17031214-40 2024-03-27T09:40:51.062 INFO:tasks.rados.rados.0.smithi170.stdout:1417: expect (ObjNum 669 snap 165 seq_num 669) 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1415: finishing write tid 2 to smithi17031214-2 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1415: finishing write tid 3 to smithi17031214-2 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: finishing write tid 4 to smithi17031214-10 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: finishing write tid 5 to smithi17031214-10 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: oid 10 updating version 0 to 75 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: oid 10 version 75 is already newer than 73 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: oid 10 updating version 75 to 76 2024-03-27T09:40:51.200 INFO:tasks.rados.rados.0.smithi170.stdout:1414: oid 10 version 76 is already newer than 75 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 76 (ObjNum 674 snap 167 seq_num 674) dirty exists 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1414: left oid 10 (ObjNum 674 snap 167 seq_num 674) 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: finishing write tid 4 to smithi17031214-2 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: finishing write tid 5 to smithi17031214-2 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: oid 2 updating version 0 to 505 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: oid 2 updating version 505 to 508 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: oid 2 version 508 is already newer than 507 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: oid 2 version 508 is already newer than 507 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 508 (ObjNum 675 snap 167 seq_num 675) dirty exists 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1415: left oid 2 (ObjNum 675 snap 167 seq_num 675) 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1416: finishing write tid 2 to smithi17031214-40 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1416: finishing write tid 3 to smithi17031214-40 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1416: finishing write tid 4 to smithi17031214-40 2024-03-27T09:40:51.201 INFO:tasks.rados.rados.0.smithi170.stdout:1416: oid 40 updating version 0 to 578 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1416: oid 40 updating version 578 to 579 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1416: oid 40 version 579 is already newer than 577 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 579 (ObjNum 676 snap 167 seq_num 676) dirty exists 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1416: left oid 40 (ObjNum 676 snap 167 seq_num 676) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1419: finishing write tid 1 to smithi17031214-6 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1419: finishing write tid 2 to smithi17031214-6 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1419: finishing write tid 3 to smithi17031214-6 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1419: finishing write tid 4 to smithi17031214-6 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1414: done (7 left) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1415: done (6 left) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1416: done (5 left) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1417: done (4 left) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1420: done (3 left) 2024-03-27T09:40:51.202 INFO:tasks.rados.rados.0.smithi170.stdout:1423: done (2 left) 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1424: delete oid 5 current snap is 167 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: finishing write tid 5 to smithi17031214-6 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: oid 6 updating version 0 to 275 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: oid 6 updating version 275 to 277 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: oid 6 updating version 277 to 278 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: oid 6 version 278 is already newer than 276 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 278 (ObjNum 678 snap 167 seq_num 678) dirty exists 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1419: left oid 6 (ObjNum 678 snap 167 seq_num 678) 2024-03-27T09:40:51.203 INFO:tasks.rados.rados.0.smithi170.stdout:1422: finishing write tid 1 to smithi17031214-50 2024-03-27T09:40:51.237 INFO:tasks.rados.rados.0.smithi170.stdout:1422: finishing write tid 2 to smithi17031214-50 2024-03-27T09:40:51.238 INFO:tasks.rados.rados.0.smithi170.stdout:1419: done (2 left) 2024-03-27T09:40:51.238 INFO:tasks.rados.rados.0.smithi170.stdout:1424: done (1 left) 2024-03-27T09:40:51.238 INFO:tasks.rados.rados.0.smithi170.stdout:1425: write (excl) oid 40 current snap is 167 2024-03-27T09:40:51.238 INFO:tasks.rados.rados.0.smithi170.stdout:1425: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-03-27T09:40:51.245 INFO:tasks.rados.rados.0.smithi170.stdout:1425: writing smithi17031214-40 from 662374 to 1137609 tid 1 2024-03-27T09:40:51.254 INFO:tasks.rados.rados.0.smithi170.stdout:1425: writing smithi17031214-40 from 1609190 to 2286082 tid 2 2024-03-27T09:40:51.264 INFO:tasks.rados.rados.0.smithi170.stdout:1425: writing smithi17031214-40 from 2812481 to 3436582 tid 3 2024-03-27T09:40:51.264 INFO:tasks.rados.rados.0.smithi170.stdout:1426: snap_remove snap 164 2024-03-27T09:40:52.047 INFO:tasks.rados.rados.0.smithi170.stdout:1426: done (2 left) 2024-03-27T09:40:52.047 INFO:tasks.rados.rados.0.smithi170.stdout:1427: write oid 1 current snap is 167 2024-03-27T09:40:52.047 INFO:tasks.rados.rados.0.smithi170.stdout:1422: finishing write tid 3 to smithi17031214-50 2024-03-27T09:40:52.047 INFO:tasks.rados.rados.0.smithi170.stdout:1427: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-03-27T09:40:52.053 INFO:tasks.rados.rados.0.smithi170.stdout:1427: writing smithi17031214-1 from 679181 to 1229665 tid 1 2024-03-27T09:40:52.058 INFO:tasks.rados.rados.0.smithi170.stdout:1427: writing smithi17031214-1 from 1667672 to 2204575 tid 2 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1427: writing smithi17031214-1 from 2839904 to 3275277 tid 3 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1428: read oid 35 snap -1 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1428: expect (ObjNum 673 snap 167 seq_num 673) 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1429: read oid 11 snap 162 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1429: expect (ObjNum 559 snap 131 seq_num 559) 2024-03-27T09:40:52.066 INFO:tasks.rados.rados.0.smithi170.stdout:1422: finishing write tid 4 to smithi17031214-50 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: finishing write tid 5 to smithi17031214-50 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: oid 50 updating version 0 to 447 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: oid 50 updating version 447 to 450 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: oid 50 version 450 is already newer than 450 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: oid 50 version 450 is already newer than 448 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 450 (ObjNum 679 snap 167 seq_num 679) dirty exists 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1422: left oid 50 (ObjNum 679 snap 167 seq_num 679) 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: finishing write tid 1 to smithi17031214-40 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: finishing write tid 2 to smithi17031214-40 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: finishing write tid 3 to smithi17031214-40 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: finishing write tid 4 to smithi17031214-40 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: finishing write tid 5 to smithi17031214-40 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: oid 40 updating version 0 to 585 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: oid 40 version 585 is already newer than 584 2024-03-27T09:40:52.067 INFO:tasks.rados.rados.0.smithi170.stdout:1425: oid 40 version 585 is already newer than 582 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1425: oid 40 version 585 is already newer than 583 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 585 (ObjNum 682 snap 167 seq_num 682) dirty exists 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1425: left oid 40 (ObjNum 682 snap 167 seq_num 682) 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1430: read oid 21 snap -1 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1430: expect deleted 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1422: done (5 left) 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1425: done (4 left) 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1431: read oid 43 snap 165 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1431: expect (ObjNum 636 snap 154 seq_num 636) 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1432: read oid 3 snap -1 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1432: expect deleted 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1433: write (excl) oid 32 current snap is 167 2024-03-27T09:40:52.068 INFO:tasks.rados.rados.0.smithi170.stdout:1433: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-03-27T09:40:52.075 INFO:tasks.rados.rados.0.smithi170.stdout:1433: writing smithi17031214-32 from 695988 to 1321721 tid 1 2024-03-27T09:40:52.085 INFO:tasks.rados.rados.0.smithi170.stdout:1433: writing smithi17031214-32 from 2009801 to 2690362 tid 2 2024-03-27T09:40:52.092 INFO:tasks.rados.rados.0.smithi170.stdout:1433: writing smithi17031214-32 from 3150974 to 3495988 tid 3 2024-03-27T09:40:52.092 INFO:tasks.rados.rados.0.smithi170.stdout:1434: write (excl) oid 9 current snap is 167 2024-03-27T09:40:52.092 INFO:tasks.rados.rados.0.smithi170.stdout:1434: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-03-27T09:40:52.099 INFO:tasks.rados.rados.0.smithi170.stdout:1434: writing smithi17031214-9 from 712795 to 1130130 tid 1 2024-03-27T09:40:52.108 INFO:tasks.rados.rados.0.smithi170.stdout:1434: writing smithi17031214-9 from 1784636 to 2325208 tid 2 2024-03-27T09:40:52.117 INFO:tasks.rados.rados.0.smithi170.stdout:1434: writing smithi17031214-9 from 2894750 to 3352667 tid 3 2024-03-27T09:40:52.117 INFO:tasks.rados.rados.0.smithi170.stdout:1435: write (excl) oid 30 current snap is 167 2024-03-27T09:40:52.117 INFO:tasks.rados.rados.0.smithi170.stdout:1435: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2024-03-27T09:40:52.126 INFO:tasks.rados.rados.0.smithi170.stdout:1435: writing smithi17031214-30 from 729602 to 1222186 tid 1 2024-03-27T09:40:52.127 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:40:52.136 INFO:tasks.rados.rados.0.smithi170.stdout:1435: writing smithi17031214-30 from 1843118 to 2527348 tid 2 2024-03-27T09:40:52.142 INFO:tasks.rados.rados.0.smithi170.stdout:1435: writing smithi17031214-30 from 3322173 to 3529602 tid 3 2024-03-27T09:40:52.142 INFO:tasks.rados.rados.0.smithi170.stdout:1436: read oid 18 snap 165 2024-03-27T09:40:52.143 INFO:tasks.rados.rados.0.smithi170.stdout:1436: expect deleted 2024-03-27T09:40:52.143 INFO:tasks.rados.rados.0.smithi170.stdout:1437: snap_create 2024-03-27T09:40:52.184 INFO:tasks.rados.rados.0.smithi170.stdout:1428: expect (ObjNum 673 snap 167 seq_num 673) 2024-03-27T09:40:52.255 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:52.252+0000 7f3a92813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:52.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:52.259+0000 7f3a92813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:52.288 INFO:teuthology.orchestra.run.smithi170.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-03-27T09:40:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-27T09:40:52.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:40:52.381 INFO:tasks.rados.rados.0.smithi170.stdout:1427: finishing write tid 1 to smithi17031214-1 2024-03-27T09:40:52.381 INFO:tasks.rados.rados.0.smithi170.stdout:1433: finishing write tid 1 to smithi17031214-32 2024-03-27T09:40:52.381 INFO:tasks.rados.rados.0.smithi170.stdout:1427: finishing write tid 2 to smithi17031214-1 2024-03-27T09:40:52.381 INFO:tasks.rados.rados.0.smithi170.stdout:1433: finishing write tid 2 to smithi17031214-32 2024-03-27T09:40:52.381 INFO:tasks.rados.rados.0.smithi170.stdout:1429: expect (ObjNum 559 snap 131 seq_num 559) 2024-03-27T09:40:52.521 INFO:tasks.rados.rados.0.smithi170.stdout:1431: expect (ObjNum 636 snap 154 seq_num 636) 2024-03-27T09:40:52.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:52.679 INFO:tasks.rados.rados.0.smithi170.stdout:1435: finishing write tid 1 to smithi17031214-30 2024-03-27T09:40:52.679 INFO:tasks.rados.rados.0.smithi170.stdout:1433: finishing write tid 3 to smithi17031214-32 2024-03-27T09:40:52.679 INFO:tasks.rados.rados.0.smithi170.stdout:1427: finishing write tid 3 to smithi17031214-1 2024-03-27T09:40:52.679 INFO:tasks.rados.rados.0.smithi170.stdout:1427: finishing write tid 4 to smithi17031214-1 2024-03-27T09:40:52.679 INFO:tasks.rados.rados.0.smithi170.stdout:1433: finishing write tid 4 to smithi17031214-32 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: finishing write tid 5 to smithi17031214-1 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: oid 1 updating version 0 to 281 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: oid 1 version 281 is already newer than 281 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: oid 1 updating version 281 to 283 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: oid 1 version 283 is already newer than 283 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 283 (ObjNum 683 snap 167 seq_num 683) dirty exists 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1427: left oid 1 (ObjNum 683 snap 167 seq_num 683) 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1433: finishing write tid 5 to smithi17031214-32 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1433: oid 32 updating version 0 to 447 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1433: oid 32 version 447 is already newer than 446 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1433: oid 32 updating version 447 to 448 2024-03-27T09:40:52.680 INFO:tasks.rados.rados.0.smithi170.stdout:1433: oid 32 version 448 is already newer than 445 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 448 (ObjNum 684 snap 167 seq_num 684) dirty exists 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1433: left oid 32 (ObjNum 684 snap 167 seq_num 684) 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1434: finishing write tid 1 to smithi17031214-9 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: finishing write tid 2 to smithi17031214-30 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: finishing write tid 3 to smithi17031214-30 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1434: finishing write tid 2 to smithi17031214-9 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: finishing write tid 4 to smithi17031214-30 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: finishing write tid 5 to smithi17031214-30 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: oid 30 updating version 0 to 305 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: oid 30 updating version 305 to 307 2024-03-27T09:40:52.681 INFO:tasks.rados.rados.0.smithi170.stdout:1435: oid 30 version 307 is already newer than 304 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1435: oid 30 version 307 is already newer than 306 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 307 (ObjNum 686 snap 167 seq_num 686) dirty exists 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1435: left oid 30 (ObjNum 686 snap 167 seq_num 686) 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: finishing write tid 3 to smithi17031214-9 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: finishing write tid 4 to smithi17031214-9 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: finishing write tid 5 to smithi17031214-9 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: oid 9 updating version 0 to 592 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: oid 9 updating version 592 to 594 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: oid 9 version 594 is already newer than 593 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: oid 9 version 594 is already newer than 591 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 594 (ObjNum 685 snap 167 seq_num 685) dirty exists 2024-03-27T09:40:52.682 INFO:tasks.rados.rados.0.smithi170.stdout:1434: left oid 9 (ObjNum 685 snap 167 seq_num 685) 2024-03-27T09:40:52.714 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:52.717 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:40:53.050 INFO:tasks.rados.rados.0.smithi170.stdout:1427: done (10 left) 2024-03-27T09:40:53.050 INFO:tasks.rados.rados.0.smithi170.stdout:1428: done (9 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1429: done (8 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1430: done (7 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1431: done (6 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1432: done (5 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1433: done (4 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1434: done (3 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1435: done (2 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1436: done (1 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1437: done (0 left) 2024-03-27T09:40:53.051 INFO:tasks.rados.rados.0.smithi170.stdout:1438: delete oid 47 current snap is 168 2024-03-27T09:40:53.107 INFO:tasks.rados.rados.0.smithi170.stdout:1438: done (0 left) 2024-03-27T09:40:53.107 INFO:tasks.rados.rados.0.smithi170.stdout:1439: write oid 27 current snap is 168 2024-03-27T09:40:53.108 INFO:tasks.rados.rados.0.smithi170.stdout:1439: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-03-27T09:40:53.114 INFO:tasks.rados.rados.0.smithi170.stdout:1439: writing smithi17031214-27 from 763216 to 1406298 tid 1 2024-03-27T09:40:53.121 INFO:tasks.rados.rados.0.smithi170.stdout:1439: writing smithi17031214-27 from 1843729 to 2531628 tid 2 2024-03-27T09:40:53.126 INFO:tasks.rados.rados.0.smithi170.stdout:1439: writing smithi17031214-27 from 3260666 to 3563216 tid 3 2024-03-27T09:40:53.126 INFO:tasks.rados.rados.0.smithi170.stdout:1440: delete oid 38 current snap is 168 2024-03-27T09:40:53.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2024-03-27T09:40:53.160 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-27T09:40:53.174 INFO:tasks.rados.rados.0.smithi170.stdout:1440: done (1 left) 2024-03-27T09:40:53.174 INFO:tasks.rados.rados.0.smithi170.stdout:1441: snap_remove snap 160 2024-03-27T09:40:53.664 INFO:tasks.ceph.osd.0:Sent signal 15 2024-03-27T09:40:53.665 INFO:tasks.ceph.osd.2:Sent signal 15 2024-03-27T09:40:53.665 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-27T09:40:53.665 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-27T09:40:53.665 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-27T09:40:53.666 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:40:53.665+0000 7f66e59ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29534) UID: 0 2024-03-27T09:40:53.666 INFO:tasks.ceph.mon.a.smithi170.stderr:2024-03-27T09:40:53.665+0000 7f66e59ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-27T09:40:53.667 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:40:53.666+0000 7f15c33f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29833) UID: 0 2024-03-27T09:40:53.667 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-27T09:40:53.666+0000 7f15c33f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-27T09:40:54.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:40:54.483 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:40:54.480+0000 7fb05246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:40:57.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:40:57.830 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:40:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:40:57.833 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:02.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:02.948 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:02.951 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:07.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:08.063 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:08.067 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:13.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:13.180 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:13.183 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:18.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:18.297 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:18.300 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:23.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:23.413 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:23.416 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:28.417 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:28.528 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:28.532 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:33.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:33.645 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:33.648 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:38.649 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:38.760 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:38.764 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:43.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:43.877 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:43.880 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-27T09:41:48.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-27T09:41:48.994 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-27T09:41:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-27T09:41:48.998 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.1 restart 2024-03-27T09:41:48.998 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_Matan-B_ceph_6db07181a4df6107bd2ae48adf3d2421c02fa2c2/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.1 restart 2024-03-27T09:42:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:42:54.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:42:56.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:42:56.522 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:42:56.519+0000 7fbea5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:44:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:44:56.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:44:58.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:44:58.563 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:44:58.561+0000 7f131691d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:46:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:46:58.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:47:00.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:47:00.607 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:47:00.604+0000 7f562e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:49:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:49:00.527 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:49:02.528 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:49:02.649 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:49:02.646+0000 7f0b1ad1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:51:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:51:02.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:51:04.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:51:04.689 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:51:04.689+0000 7f5d5155d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:53:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:53:04.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:53:06.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:53:06.727 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:53:06.730+0000 7f9084006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:55:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:55:06.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:55:08.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:55:08.765 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:55:08.770+0000 7f17d27d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:57:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:57:08.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:57:10.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T09:57:10.806 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:57:10.813+0000 7f60142ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T09:59:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T09:59:10.722 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T09:59:12.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T09:59:12.844 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T09:59:12.853+0000 7efe1b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:01:12.761 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:01:12.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:01:14.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:01:14.886 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:01:14.896+0000 7f0e03dfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:03:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:03:14.804 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:03:16.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:03:16.925 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:03:16.935+0000 7f3df306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:05:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:05:16.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:05:18.846 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:05:18.966 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:05:18.978+0000 7fc34886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:07:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:07:18.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:07:20.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:07:21.007 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:07:21.020+0000 7f744515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:09:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:09:20.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:09:22.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:09:23.051 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:09:23.064+0000 7f5acfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:11:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:11:22.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:11:24.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:11:25.092 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:11:25.107+0000 7fea52d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:13:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:13:25.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:13:27.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:13:27.129 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:13:27.144+0000 7f321839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:15:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:15:27.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:15:29.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:15:29.171 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:15:29.187+0000 7efd9e0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:17:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:17:29.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:17:31.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:17:31.209 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:17:31.226+0000 7ff543464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:19:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:19:31.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:19:33.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:19:33.254 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:19:33.271+0000 7fab80c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:21:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:21:33.170 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:21:35.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:21:35.294 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:21:35.311+0000 7f824e71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:23:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:23:35.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:23:37.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:23:37.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:23:37.352+0000 7fb8f262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:25:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:25:37.253 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:25:39.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:25:39.378 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:25:39.397+0000 7fc6b219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:27:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:27:39.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:27:41.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:27:41.416 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:27:41.436+0000 7f89e191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:29:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:29:41.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:29:43.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:29:43.460 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:29:43.480+0000 7f3ac9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:31:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:31:43.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:31:45.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:31:45.528 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:31:45.548+0000 7f49c719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:33:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:33:45.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:33:47.448 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:33:47.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:33:47.585+0000 7f3cc7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:35:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:35:47.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:35:49.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:35:49.609 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:35:49.630+0000 7f1a2359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:37:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:37:49.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:37:51.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:37:51.652 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:37:51.674+0000 7f7298a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:39:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:39:51.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:39:53.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:39:53.691 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:39:53.714+0000 7f51e9413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:41:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:41:53.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:41:55.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:41:55.732 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:41:55.755+0000 7f99a9b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:43:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:43:55.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:43:57.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:43:57.774 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:43:57.798+0000 7f8f7711d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:45:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:45:57.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:45:59.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:45:59.818 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:45:59.842+0000 7ff57222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:47:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:47:59.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:48:01.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:48:01.856 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:48:01.880+0000 7f810466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:50:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:50:01.776 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:50:03.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:50:03.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:50:03.924+0000 7f1efeead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:52:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:52:03.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:52:05.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:52:05.940 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:52:05.966+0000 7f3df099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:54:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:54:05.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:54:07.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:54:07.982 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:54:08.008+0000 7fa40a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:56:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:56:07.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:56:09.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T10:56:10.021 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:56:10.047+0000 7f3a2f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T10:58:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T10:58:09.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T10:58:11.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T10:58:12.062 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T10:58:12.089+0000 7f5c708ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:00:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:00:11.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:00:13.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:00:14.102 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:00:14.129+0000 7f64842ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:02:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:02:14.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:02:16.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:02:16.146 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:02:16.173+0000 7fd9d1c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:04:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:04:16.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:04:18.066 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:04:18.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:04:18.215+0000 7f869b5f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:06:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:06:18.106 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:06:20.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:06:20.228 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:06:20.257+0000 7fc36046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:08:20.146 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:08:20.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:08:22.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:08:22.270 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:08:22.299+0000 7fb2d3c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:10:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:10:22.189 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:10:24.190 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:10:24.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:10:24.340+0000 7f328046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:12:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:12:24.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:12:26.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:12:26.351 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:12:26.381+0000 7f47a4d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:14:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:14:26.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:14:28.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:14:28.392 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:14:28.422+0000 7fb573e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:16:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:16:28.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:16:30.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:16:30.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:16:30.466+0000 7f74e62ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:18:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:18:30.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:18:32.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:18:32.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:18:32.508+0000 7ff51422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:20:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:20:32.395 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:20:34.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:20:34.516 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:20:34.549+0000 7fe0ec26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:22:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:22:34.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:22:36.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:22:36.559 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:22:36.592+0000 7ff9d1b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:24:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:24:36.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:24:38.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:24:38.602 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:24:38.635+0000 7fd67b11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:26:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:26:38.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:26:40.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:26:40.643 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:26:40.677+0000 7f0c2ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:28:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:28:40.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:28:42.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:28:42.683 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:28:42.718+0000 7ff25091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:30:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:30:42.602 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:30:44.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:30:44.724 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:30:44.759+0000 7f8f96d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:32:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:32:44.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:32:46.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:32:46.767 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:32:46.800+0000 7f1e50eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:34:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:34:46.694 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:34:48.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:34:48.816 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:34:48.841+0000 7f404fc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:36:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:36:48.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:36:50.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:36:50.866 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:36:50.884+0000 7f7bb619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:38:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:38:50.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:38:52.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:38:52.912 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:38:52.925+0000 7f6ec0dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:40:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:40:52.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:40:54.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:40:54.957 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:40:54.965+0000 7fa66819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:42:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:42:54.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:42:56.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:42:57.003 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:42:57.008+0000 7fd42846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:44:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:44:56.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:44:58.927 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:44:59.047 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:44:59.049+0000 7fd6dd0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:46:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:46:58.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:47:00.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:47:01.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:47:01.090+0000 7f85ea35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:49:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:49:01.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:49:03.014 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:49:03.136 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:49:03.133+0000 7f70c406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:51:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:51:03.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:51:05.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:51:05.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:51:05.174+0000 7f75244ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:53:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:53:05.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:53:07.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:53:07.223 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:53:07.217+0000 7f6938cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:55:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:55:07.144 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:55:09.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:55:09.267 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:55:09.259+0000 7f48f5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:57:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:57:09.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:57:11.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T11:57:11.310 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:57:11.301+0000 7f676ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T11:59:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T11:59:11.228 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T11:59:13.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T11:59:13.350 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T11:59:13.340+0000 7f505039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:01:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:01:13.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:01:15.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:01:15.392 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:01:15.380+0000 7fa53dd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:03:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:03:15.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:03:17.314 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:03:17.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:03:17.422+0000 7f544351d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:05:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:05:17.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:05:19.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:05:19.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:05:19.463+0000 7fad8226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:07:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:07:19.395 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:07:21.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:07:21.518 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:07:21.505+0000 7f02cd75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:09:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:09:21.435 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:09:23.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:09:23.558 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:09:23.547+0000 7f15c5b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:11:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:11:23.474 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:11:25.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:11:25.596 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:11:25.586+0000 7f8f98c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:13:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:13:25.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:13:27.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:13:27.636 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:13:27.628+0000 7f0656b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:15:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:15:27.556 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:15:29.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:15:29.677 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:15:29.669+0000 7fd5a4c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:17:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:17:29.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:17:31.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:17:31.718 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:17:31.710+0000 7f57a5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:19:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:19:31.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:19:33.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:19:33.758 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:19:33.751+0000 7f04ba6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:21:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:21:33.677 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:21:35.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:21:35.799 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:21:35.792+0000 7fe862b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:23:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:23:35.719 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:23:37.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:23:37.839 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:23:37.832+0000 7f90aaa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:25:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:25:37.759 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:25:39.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:25:39.881 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:25:39.874+0000 7f655b9d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:27:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:27:39.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:27:41.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:27:41.927 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:27:41.920+0000 7faf41260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:29:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:29:41.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:29:43.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:29:43.969 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:29:43.962+0000 7f6e6859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:31:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:31:43.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:31:45.891 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:31:46.011 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:31:46.003+0000 7ff0740ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:33:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:33:45.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:33:47.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:33:48.052 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:33:48.042+0000 7f79643ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:35:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:35:47.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:35:49.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:35:50.092 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:35:50.083+0000 7fa9b6f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:37:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:37:50.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:37:52.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:37:52.133 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:37:52.125+0000 7faffdb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:39:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:39:52.057 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:39:54.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:39:54.179 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:39:54.170+0000 7f60222ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:41:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:41:54.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:41:56.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:41:56.218 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:41:56.212+0000 7f2680c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:43:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:43:56.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:43:58.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:43:58.257 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:43:58.253+0000 7f93e535d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:45:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:45:58.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:46:00.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:46:00.296 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:46:00.294+0000 7fbf9b75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:48:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:48:00.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:48:02.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:48:02.334 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:48:02.334+0000 7f28c30ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:50:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:50:02.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:50:04.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:50:04.372 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:50:04.373+0000 7f4b95aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:52:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:52:04.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:52:06.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:52:06.412 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:52:06.414+0000 7f87997e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:54:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:54:06.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:54:08.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:54:08.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:54:08.455+0000 7ff5f679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:56:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:56:08.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:56:10.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T12:56:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:56:10.496+0000 7f6bd322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T12:58:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T12:58:10.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T12:58:12.413 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T12:58:12.534 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T12:58:12.539+0000 7f96c086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:00:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:00:12.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:00:14.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:00:14.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:00:14.581+0000 7f1c27d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:02:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:02:14.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:02:16.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:02:16.617 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:02:16.622+0000 7f7d5d406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:04:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:04:16.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:04:18.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:04:18.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:04:18.664+0000 7f825d2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:06:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:06:18.577 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:06:20.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:06:20.700 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:06:20.705+0000 7f6c7ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:08:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:08:20.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:08:22.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:08:22.740 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:08:22.746+0000 7f80fdd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:10:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:10:22.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:10:24.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:10:24.782 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:10:24.788+0000 7f9cbd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:12:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:12:24.701 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:12:26.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:12:26.823 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:12:26.829+0000 7fd00b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:14:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:14:26.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:14:28.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:14:28.865 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:14:28.871+0000 7faa494ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:16:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:16:28.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:16:30.786 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:16:30.908 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:16:30.915+0000 7f2341ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:18:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:18:30.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:18:32.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:18:32.948 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:18:32.953+0000 7fe4b571d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:20:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:20:32.869 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:20:34.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:20:34.992 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:20:34.997+0000 7fcfe9024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:22:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:22:34.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:22:36.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:22:37.033 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:22:37.038+0000 7f1181bfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:24:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:24:36.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:24:38.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:24:39.075 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:24:39.080+0000 7fb7bda2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:26:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:26:38.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:26:40.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:26:41.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:26:41.121+0000 7f1a32d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:28:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:28:41.037 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:28:43.038 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:28:43.160 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:28:43.164+0000 7f72c8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:30:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:30:43.078 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:30:45.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:30:45.201 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:30:45.205+0000 7f793baad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:32:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:32:45.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:32:47.121 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:32:47.241 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:32:47.245+0000 7f4a428ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:34:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:34:47.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:34:49.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:34:49.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:34:49.290+0000 7f38a3413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:36:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:36:49.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:36:51.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:36:51.328 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:36:51.332+0000 7f241a0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:38:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:38:51.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:38:53.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:38:53.368 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:38:53.372+0000 7feeaaa13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:40:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:40:53.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:40:55.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:40:55.410 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:40:55.414+0000 7f389591d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:42:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:42:55.329 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:42:57.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:42:57.451 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:42:57.455+0000 7f364599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:44:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:44:57.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:44:59.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:44:59.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:44:59.497+0000 7f2ee20ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:46:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:46:59.413 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:47:01.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:47:01.536 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:47:01.541+0000 7f0f7691d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:49:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:49:01.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:49:03.456 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:49:03.576 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:49:03.581+0000 7fa08b99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:51:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:51:03.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:51:05.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:51:05.613 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:51:05.617+0000 7f5c36d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:53:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:53:05.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:53:07.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:53:07.657 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:53:07.662+0000 7f771f4ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:55:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:55:07.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:55:09.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:55:09.698 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:55:09.702+0000 7f2e8c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:57:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:57:09.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:57:11.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T13:57:11.740 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:57:11.744+0000 7f9693354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T13:59:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T13:59:11.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T13:59:13.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T13:59:13.781 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T13:59:13.785+0000 7f812ebf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:01:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:01:13.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:01:15.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:01:15.820 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:01:15.824+0000 7f102d22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:03:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:03:15.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:03:17.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:03:17.860 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:03:17.864+0000 7f3080c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:05:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:05:17.780 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:05:19.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:05:19.904 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:05:19.907+0000 7fee3579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:07:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:07:19.822 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:07:21.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:07:21.945 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:07:21.948+0000 7f5911a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:09:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:09:21.862 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:09:23.863 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:09:23.984 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:09:23.988+0000 7f56dedd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:11:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:11:23.904 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:11:25.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:11:26.027 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:11:26.030+0000 7f6e5af1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:13:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:13:25.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:13:27.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:13:28.067 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:13:28.071+0000 7f545cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:15:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:15:27.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:15:29.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:15:30.109 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:15:30.112+0000 7f7f58260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:17:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:17:30.027 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:17:32.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:17:32.150 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:17:32.154+0000 7fdcc6cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:19:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:19:32.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:19:34.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:19:34.192 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:19:34.196+0000 7f562895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:21:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:21:34.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:21:36.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:21:36.231 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:21:36.235+0000 7f1bbcd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:23:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:23:36.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:23:38.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:23:38.271 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:23:38.275+0000 7f99218ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:25:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:25:38.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:25:40.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:25:40.316 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:25:40.320+0000 7fc16a0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:27:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:27:40.234 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:27:42.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:27:42.356 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:27:42.360+0000 7fa418e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:29:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:29:42.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:29:44.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:29:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:29:44.400+0000 7fa16d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:31:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:31:44.317 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:31:46.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:31:46.439 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:31:46.443+0000 7f359ee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:33:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:33:46.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:33:48.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:33:48.480 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:33:48.484+0000 7fb65c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:35:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:35:48.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:35:50.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:35:50.520 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:35:50.524+0000 7f01a4a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:37:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:37:50.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:37:52.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:37:52.560 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:37:52.564+0000 7fb88419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:39:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:39:52.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:39:54.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:39:54.603 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:39:54.607+0000 7f4be599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:41:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:41:54.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:41:56.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:41:56.643 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:41:56.647+0000 7fae683ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:43:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:43:56.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:43:58.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:43:58.683 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:43:58.688+0000 7f0b6df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:45:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:45:58.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:46:00.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:46:00.725 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:46:00.729+0000 7f811239d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:48:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:48:00.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:48:02.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:48:02.766 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:48:02.770+0000 7f58afeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:50:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:50:02.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:50:04.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:50:04.808 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:50:04.812+0000 7fd64f0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:52:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:52:04.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:52:06.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:52:06.852 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:52:06.856+0000 7fcdecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:54:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:54:06.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:54:08.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:54:08.892 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:54:08.896+0000 7fa2cce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:56:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:56:08.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:56:10.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T14:56:10.931 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:56:10.935+0000 7f3544d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T14:58:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T14:58:10.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T14:58:12.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T14:58:12.974 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T14:58:12.978+0000 7f871df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:00:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:00:12.894 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:00:14.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:00:15.016 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:00:15.018+0000 7fe34166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:02:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:02:14.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:02:16.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:02:17.057 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:02:17.058+0000 7fe3f22ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:04:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:04:16.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:04:18.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:04:19.100 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:04:19.100+0000 7f2b1c5fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:06:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:06:19.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:06:21.021 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:06:21.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:06:21.141+0000 7f2b548ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:08:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:08:21.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:08:23.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:08:23.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:08:23.185+0000 7ff7b82ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:10:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:10:23.103 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:10:25.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:10:25.227 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:10:25.225+0000 7f49a126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:12:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:12:25.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:12:27.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:12:27.267 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:12:27.264+0000 7f813119d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:14:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:14:27.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:14:29.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:14:29.308 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:14:29.305+0000 7f0ab559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:16:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:16:29.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:16:31.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:16:31.349 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:16:31.345+0000 7fd587c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:18:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:18:31.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:18:33.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:18:33.390 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:18:33.386+0000 7f056dfbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:20:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:20:33.309 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:20:35.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:20:35.433 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:20:35.430+0000 7f9114154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:22:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:22:35.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:22:37.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:22:37.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:22:37.471+0000 7f029ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:24:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:24:37.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:24:39.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:24:39.516 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:24:39.512+0000 7f412a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:26:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:26:39.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:26:41.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:26:41.564 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:26:41.558+0000 7f4175eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:28:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:28:41.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:28:43.484 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:28:43.604 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:28:43.599+0000 7f068802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:30:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:30:43.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:30:45.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:30:45.646 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:30:45.640+0000 7f0e397e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:32:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:32:45.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:32:47.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:32:47.686 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:32:47.682+0000 7f0e0035d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:34:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:34:47.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:34:49.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:34:49.725 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:34:49.722+0000 7f2db2cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:36:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:36:49.643 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:36:51.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:36:51.766 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:36:51.765+0000 7f00fa26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:38:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:38:51.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:38:53.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:38:53.805 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:38:53.805+0000 7fbdfadfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:40:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:40:53.724 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:40:55.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:40:55.845 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:40:55.846+0000 7fefbb79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:42:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:42:55.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:42:57.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:42:57.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:42:57.890+0000 7fce45aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:44:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:44:57.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:44:59.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:44:59.928 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:44:59.930+0000 7f7a013fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:46:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:46:59.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:47:01.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:47:01.968 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:47:01.971+0000 7fb7c519d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:49:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:49:01.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:49:03.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:49:04.008 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:49:04.012+0000 7f4b4f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:51:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:51:03.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:51:05.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:51:06.048 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:51:06.052+0000 7f5a3cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:53:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:53:05.968 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:53:07.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:53:08.090 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:53:08.095+0000 7f855522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:55:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:55:08.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:55:10.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:55:10.130 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:55:10.135+0000 7fecae46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:57:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:57:10.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:57:12.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T15:57:12.170 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:57:12.175+0000 7ff0d775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T15:59:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T15:59:12.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T15:59:14.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T15:59:14.209 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T15:59:14.214+0000 7fd10775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:01:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:01:14.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:01:16.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:01:16.251 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:01:16.257+0000 7f02bf59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:03:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:03:16.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:03:18.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:03:18.292 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:03:18.298+0000 7f664a86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:05:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:05:18.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:05:20.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:05:20.333 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:05:20.339+0000 7fe6868ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:07:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:07:20.253 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:07:22.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:07:22.373 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:07:22.378+0000 7f6d670ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:09:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:09:22.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:09:24.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:09:24.415 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:09:24.418+0000 7ff59c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:11:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:11:24.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:11:26.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:11:26.459 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:11:26.461+0000 7fa3afe24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:13:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:13:26.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:13:28.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:13:28.500 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:13:28.501+0000 7f5e190ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:15:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:15:28.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:15:30.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:15:30.542 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:15:30.542+0000 7f630f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:17:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:17:30.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:17:32.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:17:32.584 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:17:32.584+0000 7f54f6417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:19:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:19:32.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:19:34.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:19:34.622 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:19:34.622+0000 7fa7985d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:21:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:21:34.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:21:36.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:21:36.663 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:21:36.662+0000 7f087f2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:23:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:23:36.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:23:38.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:23:38.705 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:23:38.704+0000 7f7d5959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:25:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:25:38.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:25:40.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:25:40.745 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:25:40.743+0000 7fa19fd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:27:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:27:40.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:27:42.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:27:42.788 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:27:42.786+0000 7f80602ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:29:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:29:42.706 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:29:44.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:29:44.829 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:29:44.827+0000 7f930fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:31:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:31:44.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:31:46.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:31:46.871 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:31:46.868+0000 7f88d8954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:33:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:33:46.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:33:48.791 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:33:48.913 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:33:48.911+0000 7fa720417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:35:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:35:48.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:35:50.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:35:50.953 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:35:50.951+0000 7fd6bb4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:37:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:37:50.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:37:52.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:37:52.994 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:37:52.991+0000 7f7918264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:39:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:39:52.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:39:54.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:39:55.035 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:39:55.032+0000 7fe4e382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:41:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:41:54.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:41:56.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:41:57.075 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:41:57.073+0000 7fc37579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:43:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:43:56.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:43:58.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:43:59.116 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:43:59.113+0000 7f5377b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:45:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:45:59.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:46:01.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:46:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:46:01.153+0000 7f2c34e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:48:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:48:01.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:48:03.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:48:03.196 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:48:03.194+0000 7fd0232ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:50:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:50:03.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:50:05.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:50:05.238 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:50:05.235+0000 7fc4ca75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:52:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:52:05.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:52:07.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:52:07.279 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:52:07.277+0000 7fd92246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:54:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:54:07.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:54:09.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:54:09.320 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:54:09.318+0000 7fe82419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:56:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:56:09.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:56:11.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T16:56:11.359 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:56:11.357+0000 7f6c1906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T16:58:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T16:58:11.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T16:58:13.280 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T16:58:13.399 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T16:58:13.397+0000 7fe409dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:00:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:00:13.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:00:15.325 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:00:15.446 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:00:15.444+0000 7fc64a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:02:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:02:15.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:02:17.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:02:17.487 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:02:17.484+0000 7fa3dd864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:04:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:04:17.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:04:19.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:04:19.527 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:04:19.524+0000 7f71593bf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:06:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:06:19.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:06:21.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:06:21.570 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:06:21.565+0000 7f6dd182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:08:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:08:21.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:08:23.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:08:23.610 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:08:23.607+0000 7f5677813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:10:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:10:23.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:10:25.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:10:25.650 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:10:25.647+0000 7f4b5b1cf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:12:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:12:25.570 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:12:27.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:12:27.692 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:12:27.689+0000 7fc843b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:14:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:14:27.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:14:29.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:14:29.732 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:14:29.730+0000 7f7c20cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:16:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:16:29.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:16:31.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:16:31.773 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:16:31.770+0000 7fc44cfbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:18:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:18:31.693 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:18:33.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:18:33.816 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:18:33.813+0000 7febfbc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:20:33.732 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:20:33.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:20:35.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:20:35.855 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:20:35.853+0000 7f06a7f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:22:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:22:35.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:22:37.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:22:37.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:22:37.895+0000 7f10c6217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:24:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:24:37.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:24:39.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:24:39.938 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:24:39.935+0000 7f0e5e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:26:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:26:39.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:26:41.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:26:41.979 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:26:41.977+0000 7fb815b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:28:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:28:41.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:28:43.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:28:44.022 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:28:44.019+0000 7f91a235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:30:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:30:43.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:30:45.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:30:46.063 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:30:46.060+0000 7fe7c439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:32:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:32:45.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:32:47.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:32:48.104 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:32:48.101+0000 7fcc190ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:34:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:34:48.023 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:34:50.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:34:50.144 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:34:50.141+0000 7f3897ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:36:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:36:50.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:36:52.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:36:52.186 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:36:52.183+0000 7f5ecea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:38:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:38:52.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:38:54.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:38:54.225 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:38:54.223+0000 7f4e0b224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:40:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:40:54.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:40:56.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:40:56.267 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:40:56.264+0000 7f75b6fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:42:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:42:56.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:42:58.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:42:58.307 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:42:58.304+0000 7f1ac12ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:44:58.225 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:44:58.226 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:45:00.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:45:00.348 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:45:00.345+0000 7f211711d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:47:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:47:00.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:47:02.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:47:02.386 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:47:02.384+0000 7f1967a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:49:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:49:02.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:49:04.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:49:04.429 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:49:04.427+0000 7fc035d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:51:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:51:04.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:51:06.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:51:06.470 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:51:06.467+0000 7f27ae024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:53:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:53:06.387 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:53:08.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:53:08.510 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:53:08.507+0000 7fb7c482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:55:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:55:08.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:55:10.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:55:10.550 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:55:10.548+0000 7f6df6817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:57:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:57:10.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:57:12.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T17:57:12.590 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:57:12.588+0000 7f4b8af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T17:59:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T17:59:12.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T17:59:14.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T17:59:14.631 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T17:59:14.629+0000 7fa5775fa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:01:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:01:14.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:01:16.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:01:16.672 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:01:16.670+0000 7f42d706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:03:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:03:16.592 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:03:18.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:03:18.715 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:03:18.713+0000 7fd0f5d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:05:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:05:18.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:05:20.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:05:20.755 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:05:20.753+0000 7f450539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:07:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:07:20.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:07:22.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:07:22.797 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:07:22.796+0000 7f2c1ebbf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:09:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:09:22.714 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:09:24.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:09:24.834 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:09:24.832+0000 7fabd299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:11:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:11:24.754 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:11:26.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:11:26.876 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:11:26.874+0000 7f45d179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:13:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:13:26.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:13:28.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:13:28.917 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:13:28.916+0000 7f676df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:15:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:15:28.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:15:30.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:15:30.957 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:15:30.955+0000 7fd7a7aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:17:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:17:30.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:17:32.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:17:32.997 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:17:32.996+0000 7ffa2811d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:19:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:19:32.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:19:34.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:19:35.039 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:19:35.038+0000 7f081586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:21:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:21:34.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:21:36.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:21:37.080 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:21:37.079+0000 7fe58e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:23:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:23:37.000 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:23:39.001 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:23:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:23:39.119+0000 7fc9dd26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:25:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:25:39.041 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:25:41.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:25:41.161 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:25:41.160+0000 7f5a0e8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:27:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:27:41.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:27:43.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:27:43.202 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:27:43.201+0000 7f733406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:29:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:29:43.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:29:45.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:29:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:29:45.241+0000 7f620635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:31:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:31:45.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:31:47.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:31:47.285 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:31:47.283+0000 7f23e051d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:33:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:33:47.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:33:49.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:33:49.327 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:33:49.325+0000 7f8c59c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:35:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:35:49.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:35:51.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:35:51.367 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:35:51.365+0000 7fd2a306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:37:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:37:51.286 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:37:53.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:37:53.407 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:37:53.405+0000 7f5a3afff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:39:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:39:53.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:39:55.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:39:55.450 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:39:55.448+0000 7f5199f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:41:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:41:55.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:41:57.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:41:57.490 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:41:57.489+0000 7ff41a11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:43:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:43:57.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:43:59.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:43:59.529 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:43:59.527+0000 7fb46b95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:45:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:45:59.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:46:01.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:46:01.572 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:46:01.570+0000 7fc1847ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:48:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:48:01.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:48:03.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:48:03.612 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:48:03.610+0000 7f6b3fb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:50:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:50:03.531 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:50:05.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:50:05.652 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:50:05.651+0000 7fbb0f860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:52:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:52:05.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:52:07.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:52:07.695 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:52:07.693+0000 7fd2d4617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:54:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:54:07.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:54:09.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:54:09.736 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:54:09.735+0000 7f8baef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:56:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:56:09.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:56:11.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T18:56:11.777 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:56:11.775+0000 7fe0084ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T18:58:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T18:58:11.695 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T18:58:13.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T18:58:13.818 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T18:58:13.815+0000 7fb768e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:00:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:00:13.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:00:15.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:00:15.857 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:00:15.855+0000 7f1feb224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:02:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:02:15.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:02:17.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:02:17.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:02:17.896+0000 7f163742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:04:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:04:17.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:04:19.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:04:19.940 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:04:19.938+0000 7f4f1535d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:06:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:06:19.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:06:21.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:06:21.981 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:06:21.979+0000 7f44a1d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:08:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:08:21.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:08:23.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:08:24.020 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:08:24.018+0000 7f5f6915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:10:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:10:23.940 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:10:25.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:10:26.062 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:10:26.060+0000 7feccbead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:12:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:12:25.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:12:27.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:12:28.104 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:12:28.102+0000 7f528ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:14:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:14:28.023 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:14:30.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:14:30.145 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:14:30.143+0000 7fd604aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:16:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:16:30.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:16:32.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:16:32.185 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:16:32.183+0000 7fbf0d6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:18:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:18:32.104 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:18:34.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:18:34.226 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:18:34.224+0000 7f1083f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:20:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:20:34.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:20:36.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:20:36.267 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:20:36.265+0000 7f32a375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:22:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:22:36.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:22:38.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:22:38.309 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:22:38.306+0000 7f278659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:24:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:24:38.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:24:40.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:24:40.349 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:24:40.347+0000 7f055cb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:26:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:26:40.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:26:42.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:26:42.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:26:42.385+0000 7f1342a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:28:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:28:42.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:28:44.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:28:44.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:28:44.429+0000 7f90f58ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:30:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:30:44.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:30:46.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:30:46.471 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:30:46.469+0000 7f542ed54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:32:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:32:46.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:32:48.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:32:48.512 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:32:48.510+0000 7f1d2899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:34:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:34:48.431 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:34:50.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:34:50.552 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:34:50.550+0000 7fbabf8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:36:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:36:50.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:36:52.472 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:36:52.591 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:36:52.589+0000 7f322bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:38:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:38:52.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:38:54.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:38:54.634 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:38:54.632+0000 7f6d14d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:40:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:40:54.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:40:56.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:40:56.674 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:40:56.672+0000 7f093e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:42:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:42:56.594 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:42:58.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:42:58.717 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:42:58.715+0000 7f694b19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:44:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:44:58.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:45:00.636 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:45:00.757 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:45:00.755+0000 7f99b751d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:47:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:47:00.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:47:02.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:47:02.797 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:47:02.795+0000 7f673a754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:49:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:49:02.715 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:49:04.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:49:04.837 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:49:04.834+0000 7fe5f1d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:51:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:51:04.756 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:51:06.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:51:06.879 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:51:06.876+0000 7f6e5b7d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:53:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:53:06.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:53:08.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:53:08.919 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:53:08.916+0000 7fa5f766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:55:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:55:08.838 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:55:10.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:55:10.962 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:55:10.960+0000 7faff255d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:57:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:57:10.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:57:12.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T19:57:13.000 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:57:12.998+0000 7f12f9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T19:59:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T19:59:12.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T19:59:14.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T19:59:15.042 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T19:59:15.040+0000 7f70a042d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:01:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:01:14.962 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:01:16.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:01:17.083 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:01:17.080+0000 7f2ec2aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:03:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:03:17.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:03:19.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:03:19.125 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:03:19.123+0000 7efc8ccad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:05:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:05:19.044 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:05:21.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:05:21.165 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:05:21.163+0000 7f3e2646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:07:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:07:21.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:07:23.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:07:23.207 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:07:23.205+0000 7f275e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:09:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:09:23.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:09:25.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:09:25.246 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:09:25.244+0000 7f7442a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:11:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:11:25.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:11:27.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:11:27.286 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:11:27.285+0000 7f7baee24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:13:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:13:27.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:13:29.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:13:29.326 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:13:29.326+0000 7fb5f839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:15:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:15:29.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:15:31.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:15:31.367 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:15:31.367+0000 7f579b22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:17:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:17:31.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:17:33.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:17:33.407 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:17:33.407+0000 7fdd0b46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:19:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:19:33.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:19:35.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:19:35.448 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:19:35.448+0000 7f653ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:21:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:21:35.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:21:37.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:21:37.491 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:21:37.492+0000 7f8fb6dd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:23:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:23:37.407 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:23:39.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:23:39.528 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:23:39.529+0000 7fe6d462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:25:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:25:39.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:25:41.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:25:41.571 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:25:41.572+0000 7f59c291d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:27:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:27:41.488 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:27:43.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:27:43.610 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:27:43.611+0000 7f49bf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:29:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:29:43.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:29:45.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:29:45.651 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:29:45.652+0000 7fe49ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:31:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:31:45.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:31:47.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:31:47.691 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:31:47.692+0000 7fdfcd26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:33:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:33:47.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:33:49.610 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:33:49.730 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:33:49.732+0000 7fe38482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:35:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:35:49.649 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:35:51.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:35:51.772 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:35:51.773+0000 7f36ae35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:37:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:37:51.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:37:53.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:37:53.813 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:37:53.815+0000 7fdca3cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:39:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:39:53.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:39:55.733 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:39:55.853 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:39:55.855+0000 7fa727a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:41:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:41:55.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:41:57.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:41:57.895 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:41:57.896+0000 7fd7b9264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:43:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:43:57.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:43:59.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:43:59.937 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:43:59.937+0000 7f2bc8264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:45:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:45:59.857 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:46:01.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:46:01.978 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:46:01.977+0000 7f300526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:48:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:48:01.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:48:03.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:48:04.020 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:48:04.019+0000 7f4fb9064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:50:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:50:03.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:50:05.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:50:06.061 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:50:06.059+0000 7ff71e620640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:52:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:52:05.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:52:07.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:52:08.103 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:52:08.101+0000 7f4375e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:54:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:54:08.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:54:10.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:54:10.143 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:54:10.141+0000 7f247eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:56:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:56:10.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:56:12.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T20:56:12.183 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:56:12.180+0000 7fc018ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T20:58:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T20:58:12.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T20:58:14.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T20:58:14.222 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T20:58:14.220+0000 7f8e8a00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:00:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:00:14.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:00:16.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:00:16.266 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:00:16.263+0000 7f3fcb46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:02:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:02:16.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:02:18.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:02:18.307 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:02:18.304+0000 7fe03f79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:04:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:04:18.225 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:04:20.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:04:20.345 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:04:20.342+0000 7f2a9a8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:06:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:06:20.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:06:22.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:06:22.387 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:06:22.383+0000 7f03caf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:08:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:08:22.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:08:24.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:08:24.430 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:08:24.427+0000 7fc658f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:10:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:10:24.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:10:26.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:10:26.469 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:10:26.466+0000 7f828c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:12:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:12:26.388 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:12:28.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:12:28.510 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:12:28.507+0000 7f7ed00ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:14:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:14:28.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:14:30.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:14:30.549 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:14:30.547+0000 7f498cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:16:30.463 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:16:30.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:16:32.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:16:32.587 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:16:32.588+0000 7f1cbb2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:18:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:18:32.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:18:34.505 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:18:34.626 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:18:34.629+0000 7ff56115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:20:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:20:34.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:20:36.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-27T21:20:36.663 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:20:36.668+0000 7f5b306ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:22:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-27T21:22:36.585 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-27T21:22:38.586 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-27T21:22:38.707 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-27T21:22:38.713+0000 7fc601d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T21:23:27.656 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-27T21:23:27.657 DEBUG:teuthology.task.console_log:Killing console logger for smithi170 2024-03-27T21:23:27.658 DEBUG:teuthology.task.console_log:Killing console logger for smithi172 2024-03-27T21:23:27.658 DEBUG:teuthology.exit:Finished running handlers