2023-12-28T12:00:53.917 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2023-12-28T12:00:53.918 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rfriedma@teuthology --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/set-chunks-read} --archive /home/teuthworker/archive/rfriedma-2023-12-28_11:36:01-rados:thrash-wip-rf-be-error-distro-default-smithi/7502838 --name rfriedma-2023-12-28_11:36:01-rados:thrash-wip-rf-be-error-distro-default-smithi -- /home/teuthworker/archive/rfriedma-2023-12-28_11:36:01-rados:thrash-wip-rf-be-error-distro-default-smithi/7502838/orig.config.yaml 2023-12-28T12:00:53.956 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-28T12:00:54.031 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rfriedma-2023-12-28_11:36:01-rados:thrash-wip-rf-be-error-distro-default-smithi/7502838 branch: wip-rf-be-error description: rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/set-chunks-read} email: null first_in_suite: false job_id: '7502838' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rfriedma-2023-12-28_11:36:01-rados:thrash-wip-rf-be-error-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rf-be-error ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd inject bad map crc probability: 0.1 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 173c344563235b95e606b93f2ce772bfeb9e148c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 173c344563235b95e606b93f2ce772bfeb9e148c thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rf-be-error sha1: 173c344563235b95e606b93f2ce772bfeb9e148c owner: scheduled_rfriedma@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6375 sha1: 173c344563235b95e606b93f2ce772bfeb9e148c sleep_before_teardown: 0 subset: 1/10 suite: rados:thrash suite_branch: wip-rf-be-error suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 173c344563235b95e606b93f2ce772bfeb9e148c targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKzBmBJ8CcZvMOx1/rBSjj8/CG+fjwjBv/2j2TU6NRCVKo4XcmdxJmGFpHr2urxSWKuRqaw23wwWKoocTB7DRU= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJZ77mtXvWefj6vPwHTp3fswdeleMzd6oCbyQe3w4tURH9MsZtXHWMkJXrsfmC/z9SZA4vGtqQqkieeM/ObkvUc= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 low_tier_pool: low_tier objects: 300 op_weights: chunk_read: 100 tier_promote: 10 ops: 4000 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 01dd171a561ca33631bf0a38b105879779f72410 timestamp: 2023-12-28_11:36:01 tube: smithi user: rfriedma verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-28T12:00:54.031 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c/qa; will attempt to use it 2023-12-28T12:00:54.032 INFO:teuthology.run:Tasks not found; will attempt to fetch 2023-12-28T12:00:54.032 INFO:teuthology.repo_utils:Cloning https://git.ceph.com/ceph-ci.git wip-rf-be-error from upstream 2023-12-28T12:01:48.522 ERROR:teuthology.repo_utils:Cloning into '/home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c'... error: unable to write file /home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c/.git/objects/pack/pack-63b1812b4aee6a440c584a7ea4ed45ecbd878be2.pack: No such file or directory fatal: cannot store pack file fatal: index-pack failed 2023-12-28T12:01:48.522 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 119, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 177, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2023-12-28T12:01:58.533 INFO:teuthology.repo_utils:Cloning https://git.ceph.com/ceph-ci.git wip-rf-be-error from upstream 2023-12-28T12:02:47.851 ERROR:teuthology.repo_utils:Cloning into '/home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c'... fatal: did not receive expected object 7aebc096ea8f9744b7bc33023d8a75129b726583 fatal: index-pack failed 2023-12-28T12:02:47.851 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 119, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 177, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2023-12-28T12:02:57.861 INFO:teuthology.repo_utils:Cloning https://git.ceph.com/ceph-ci.git wip-rf-be-error from upstream 2023-12-28T12:03:46.374 ERROR:teuthology.repo_utils:Cloning into '/home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c'... fatal: did not receive expected object 7aebc096ea8f9744b7bc33023d8a75129b726583 fatal: index-pack failed 2023-12-28T12:03:46.375 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 119, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 177, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2023-12-28T12:03:56.386 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-c_173c344563235b95e606b93f2ce772bfeb9e148c to 173c344563235b95e606b93f2ce772bfeb9e148c 2023-12-28T12:03:56.395 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 319, in reset_repo subprocess.check_output( File "/usr/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '('git', 'reset', '--hard', '173c344563235b95e606b93f2ce772bfeb9e148c')' returned non-zero exit status 128. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/virtualenv/bin/teuthology", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/run.py", line 388, in main config["suite_path"] = fetch_tasks_if_needed(config) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/run.py", line 90, in fetch_tasks_if_needed fetch_qa_suite(suite_branch, commit=suite_sha1), File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 426, in fetch_qa_suite return fetch_repo(config.get_ceph_qa_suite_git_url(), File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 128, in enforce_repo_state reset_repo(repo_url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/repo_utils.py", line 325, in reset_repo raise CommitNotFoundError(commit, repo_url) teuthology.exceptions.CommitNotFoundError: '173c344563235b95e606b93f2ce772bfeb9e148c' not found in repo: https://git.ceph.com/ceph-ci.git!