2024-02-15T12:00:42.763 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-15T12:00:42.763 DEBUG:teuthology.run:Teuthology command: teuthology --name dis-2024-02-15_00:09:12-rbd-wip-dis-testing-distro-default-smithi --description rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-stupid 4-supported-random-distro$/{ubuntu_latest} 5-data-pool/ec 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} --owner scheduled_dis@teuthology --verbose --archive /home/teuthworker/archive/dis-2024-02-15_00:09:12-rbd-wip-dis-testing-distro-default-smithi/7560581 -- /home/teuthworker/archive/dis-2024-02-15_00:09:12-rbd-wip-dis-testing-distro-default-smithi/7560581/orig.config.yaml 2024-02-15T12:00:42.807 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-15T12:00:42.894 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-02-15_00:09:12-rbd-wip-dis-testing-distro-default-smithi/7560581 branch: wip-dis-testing description: rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-stupid 4-supported-random-distro$/{ubuntu_latest} 5-data-pool/ec 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} email: idryomov@gmail.com exclude_arch: armv7l first_in_suite: false io_workload: sequential: - qemu: client.0: clone: true disks: - action: none image_name: client.0.0 - action: none image_name: client.0.1 - action: none image_name: client.0.2 test: qa/run_xfstests_qemu.sh type: block job_id: '7560581' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi migrate_workload: sequential: - exec: client.0: - sleep 60 - rbd migration execute client.0.0 - sleep 60 - rbd migration commit client.0.0 - sleep 60 - rbd migration execute client.0.1 name: dis-2024-02-15_00:09:12-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 30000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd default data pool: datapool global: mon warn on pool no app: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 enable experimental unrecoverable data corrupting features: '*' mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug randomize hobject sort order: false osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 01bbae032a07369ea5f8aa05b8893656bb22a284 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 01bbae032a07369ea5f8aa05b8893656bb22a284 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: 01bbae032a07369ea5f8aa05b8893656bb22a284 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.y - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 seed: 1588 sha1: 01bbae032a07369ea5f8aa05b8893656bb22a284 sleep_before_teardown: 0 subset: 83/99 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 01bbae032a07369ea5f8aa05b8893656bb22a284 targets: smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxLYHoy6rNB0cDgAeZasciqoiRWNeeNG3AGAABKF2eAfyhc9HOSCvQ7Z+rbHwzFXFni7W+f4gD3CTwozZHep6E= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDxQlavqMpuyJm+HdZchJlaDEdZwCN+bvMN4w8dUh5yaq+3sf/LUU9iFqT5lAJ9m5z2vgEsIZy0Em9OZMLO28OU= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYn6hmfcpVNA97dyqSjLoJP70mFdAlKEyZsYBTMyqv52N8osyA9+N/qLKa4jvSIOnPrLRLu/av+yikEx4v3qcE= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2 - sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile - sudo ceph osd pool set datapool allow_ec_overwrites true - rbd pool init datapool - exec: client.0: - mkdir /home/ubuntu/cephtest/migration - wget -nv -O /home/ubuntu/cephtest/migration/base.client.0.qcow2 http://download.ceph.com/qa/ubuntu-12.04.qcow2 - qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/migration/base.client.0.qcow2 /home/ubuntu/cephtest/migration/base.client.0.raw - dd if=/dev/zero of=/home/ubuntu/cephtest/migration/empty.raw count=1 bs=1G - echo '{"type":"raw","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/base.client.0.raw"}}' | rbd migration prepare --import-only --source-spec-path - client.0.0 - rbd migration prepare --import-only --source-spec '{"type":"raw","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.raw"}}' client.0.1 - rbd migration prepare --import-only --source-spec '{"type":"raw","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.raw"}}' client.0.2 - parallel: - io_workload - migrate_workload - exec: client.0: - rm -rf /home/ubuntu/cephtest/migration teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-15_00:09:12 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-15T12:00:42.894 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284/qa; will attempt to use it 2024-02-15T12:00:42.894 INFO:teuthology.run:Tasks not found; will attempt to fetch 2024-02-15T12:00:42.895 INFO:teuthology.repo_utils:Cloning https://github.com/ceph/ceph-ci wip-dis-testing from upstream 2024-02-15T12:01:30.176 ERROR:teuthology.repo_utils:Cloning into '/home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284'... error: unable to write file /home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284/.git/objects/pack/pack-83691cd512efc46961c973eb466d61024e37d022.pack: No such file or directory fatal: cannot store pack file fatal: index-pack failed 2024-02-15T12:01:30.177 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 119, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 177, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2024-02-15T12:01:40.188 INFO:teuthology.repo_utils:Cloning https://github.com/ceph/ceph-ci wip-dis-testing from upstream 2024-02-15T12:01:43.414 ERROR:teuthology.repo_utils:Cloning into '/home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284'... fatal: --stdin requires a git repository fatal: index-pack failed 2024-02-15T12:01:43.415 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 119, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 177, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2024-02-15T12:01:53.426 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/github.com_ceph_ceph-c_01bbae032a07369ea5f8aa05b8893656bb22a284 to 01bbae032a07369ea5f8aa05b8893656bb22a284 2024-02-15T12:01:53.461 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 319, in reset_repo subprocess.check_output( File "/usr/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '('git', 'reset', '--hard', '01bbae032a07369ea5f8aa05b8893656bb22a284')' returned non-zero exit status 128. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run.py", line 388, in main config["suite_path"] = fetch_tasks_if_needed(config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run.py", line 90, in fetch_tasks_if_needed fetch_qa_suite(suite_branch, commit=suite_sha1), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 426, in fetch_qa_suite return fetch_repo(config.get_ceph_qa_suite_git_url(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 365, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 128, in enforce_repo_state reset_repo(repo_url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/repo_utils.py", line 325, in reset_repo raise CommitNotFoundError(commit, repo_url) teuthology.exceptions.CommitNotFoundError: '01bbae032a07369ea5f8aa05b8893656bb22a284' not found in repo: https://github.com/ceph/ceph-ci!