2023-07-09T12:01:11.747 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-09T12:01:11.749 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} --archive /home/teuthworker/archive/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330382 --name yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330382/orig.config.yaml 2023-07-09T12:01:11.785 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-09T12:01:11.883 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi/7330382 branch: reef description: rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7330382' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-07-08_17:45:15-rados-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: 2a555d3f656c6f10449c1c0fa097caa31f4056c5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 2a555d3f656c6f10449c1c0fa097caa31f4056c5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: 2a555d3f656c6f10449c1c0fa097caa31f4056c5 owner: scheduled_yuriw@teuthology priority: 95 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3936 sha1: 2a555d3f656c6f10449c1c0fa097caa31f4056c5 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_2a555d3f656c6f10449c1c0fa097caa31f4056c5/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2a555d3f656c6f10449c1c0fa097caa31f4056c5 targets: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: null - ceph: log-ignorelist: - but it is still running - had wrong client addr - had wrong cluster addr - must scrub before tier agent can activate - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping updates - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SMALLER_PG_NUM\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(FS_WITH_FAILED_MDS\) - \(FS_DEGRADED\) - \(POOL_BACKFILLFULL\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(POOL_NEARFULL\) - \(POOL_APP_NOT_ENABLED\) - \(AUTH_BAD_CAPS\) - \(FS_INLINE_DATA_DEPRECATED\) - \(MON_DOWN\) - \(SLOW_OPS\) - slow request - workunit: clients: all: - cephtool - mon/pool_ops.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-08_17:45:15 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-09T12:01:11.883 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_2a555d3f656c6f10449c1c0fa097caa31f4056c5/qa; will attempt to use it 2023-07-09T12:01:11.884 INFO:teuthology.run:Tasks not found; will attempt to fetch 2023-07-09T12:01:11.885 INFO:teuthology.repo_utils:/home/teuthworker/src/github.com_ceph_ceph-c_2a555d3f656c6f10449c1c0fa097caa31f4056c5 was just updated or references a specific commit; assuming it is current 2023-07-09T12:01:11.885 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/github.com_ceph_ceph-c_2a555d3f656c6f10449c1c0fa097caa31f4056c5 to 2a555d3f656c6f10449c1c0fa097caa31f4056c5 2023-07-09T12:01:11.897 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/repo_utils.py", line 319, in reset_repo subprocess.check_output( File "/usr/lib/python3.8/subprocess.py", line 411, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '('git', 'reset', '--hard', '2a555d3f656c6f10449c1c0fa097caa31f4056c5')' returned non-zero exit status 128. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/bin/teuthology", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run.py", line 388, in main config["suite_path"] = fetch_tasks_if_needed(config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run.py", line 90, in fetch_tasks_if_needed fetch_qa_suite(suite_branch, commit=suite_sha1), File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/repo_utils.py", line 426, in fetch_qa_suite return fetch_repo(config.get_ceph_qa_suite_git_url(), File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/repo_utils.py", line 128, in enforce_repo_state reset_repo(repo_url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/repo_utils.py", line 325, in reset_repo raise CommitNotFoundError(commit, repo_url) teuthology.exceptions.CommitNotFoundError: '2a555d3f656c6f10449c1c0fa097caa31f4056c5' not found in repo: https://github.com/ceph/ceph-ci.git!