2024-04-22T10:05:12.347 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T10:05:12.348 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} --archive /home/teuthworker/archive/teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi/7666997 --owner scheduled_teuthology@teuthology --name teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi/7666997/orig.config.yaml 2024-04-22T10:05:12.394 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T10:05:12.495 INFO:teuthology.run:Config: arch: x86_64 archive_path: /home/teuthworker/archive/teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi/7666997 branch: main description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} email: ceph-qa@ceph.com first_in_suite: false job_id: '7666997' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_async_recovery_min_cost: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 30 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: d5c8530042f7705c6e8ec83a32e0ea1be9cbad40 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d5c8530042f7705c6e8ec83a32e0ea1be9cbad40 roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 226cba1c5219f5bb8d0cd056721711c87d9bf2fd owner: scheduled_teuthology@teuthology priority: 951 repo: https://git.ceph.com/ceph.git seed: 8858 sha1: d5c8530042f7705c6e8ec83a32e0ea1be9cbad40 sleep_before_teardown: 0 subset: 15139/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 226cba1c5219f5bb8d0cd056721711c87d9bf2fd tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 map_discontinuity_sleep_time: 200 min_in: 2 timeout: 1800 - rados: clients: - client.0 ec_pool: true erasure_code_crush: id: 86 max_size: 6 min_size: 3 name: jerasure86crush steps: - set_chooseleaf_tries 5 - set_choose_tries 100 - take default class hdd - choose indep 4 type host - choose indep 4 type osd - emit type: erasure erasure_code_profile: crush-failure-domain: osd k: 8 m: 6 name: jerasure86profile plugin: jerasure technique: reed_sol_van max_in_flight: 64 max_seconds: 600 objects: 1024 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400000 size: 16384 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_20:00:14 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T10:05:12.495 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa; will attempt to use it 2024-04-22T10:05:12.496 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks 2024-04-22T10:05:12.496 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T10:05:12.498 INFO:teuthology.task.internal:Checking packages... 2024-04-22T10:05:12.524 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40' 2024-04-22T10:05:12.524 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T10:05:12.524 INFO:teuthology.packaging:ref: None 2024-04-22T10:05:12.524 INFO:teuthology.packaging:tag: None 2024-04-22T10:05:12.524 INFO:teuthology.packaging:branch: main 2024-04-22T10:05:12.524 INFO:teuthology.packaging:sha1: d5c8530042f7705c6e8ec83a32e0ea1be9cbad40 2024-04-22T10:05:12.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-22T10:05:12.878 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3162.gf25efe27 2024-04-22T10:05:12.879 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T10:05:12.887 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T10:05:12.887 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T10:05:12.932 INFO:teuthology.task.internal:Saving configuration 2024-04-22T10:05:12.947 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T10:05:12.958 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T10:05:12.999 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T10:05:12.999 INFO:teuthology.task.kernel:config {}, timeout 300 2024-04-22T10:05:13.000 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T10:05:13.016 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T10:05:13.041 INFO:teuthology.task.internal:Starting timer... 2024-04-22T10:05:13.041 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T10:05:13.067 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T10:05:13.084 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T10:05:13.084 INFO:teuthology.task.ansible:There are no remotes; skipping playbook execution 2024-04-22T10:05:13.084 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T10:05:13.094 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T10:05:13.094 INFO:teuthology.run_tasks:Running task install... 2024-04-22T10:05:13.116 DEBUG:teuthology.task.install:project ceph 2024-04-22T10:05:13.116 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40'}} 2024-04-22T10:05:13.116 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40'} 2024-04-22T10:05:13.116 INFO:teuthology.task.install:Using flavor: default 2024-04-22T10:05:13.123 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T10:05:13.123 INFO:teuthology.task.install:extra packages: [] 2024-04-22T10:05:13.123 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T10:05:13.123 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T10:05:13.124 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T10:05:13.124 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T10:05:13.124 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T10:05:13.224 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T10:05:13.224 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T10:05:13.224 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T10:05:13.229 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T10:05:13.229 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T10:05:13.229 INFO:tasks.ceph:ctx.config {'arch': 'x86_64', 'archive_path': '/home/teuthworker/archive/teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi/7666997', 'branch': 'main', 'description': 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7666997', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-21_20:00:14-rados-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40'}}, 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '226cba1c5219f5bb8d0cd056721711c87d9bf2fd'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 951, 'repo': 'https://git.ceph.com/ceph.git', 'seed': 8858, 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40', 'sleep_before_teardown': 0, 'subset': '15139/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '226cba1c5219f5bb8d0cd056721711c87d9bf2fd', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.add_remotes': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.archive_upload': None}, {'internal.timer': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd5c8530042f7705c6e8ec83a32e0ea1be9cbad40', 'cluster': 'ceph'}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_crush': {'id': 86, 'max_size': 6, 'min_size': 3, 'name': 'jerasure86crush', 'steps': ['set_chooseleaf_tries 5', 'set_choose_tries 100', 'take default class hdd', 'choose indep 4 type host', 'choose indep 4 type osd', 'emit'], 'type': 'erasure'}, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 8, 'm': 6, 'name': 'jerasure86profile', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'max_in_flight': 64, 'max_seconds': 600, 'objects': 1024, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400000, 'size': 16384, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-21_20:00:14', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T10:05:13.230 INFO:tasks.ceph:remote_to_roles_to_devs: {} 2024-04-22T10:05:13.230 INFO:tasks.ceph:Generating config... 2024-04-22T10:05:13.230 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 693, in cluster mons = get_mons( File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 510, in get_mons assert mons AssertionError 2024-04-22T10:05:13.231 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T10:05:13.232 INFO:tasks.ceph:Compressing logs... 2024-04-22T10:05:13.232 INFO:tasks.ceph:Archiving logs... 2024-04-22T10:05:13.232 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 693, in cluster mons = get_mons( File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 510, in get_mons assert mons AssertionError 2024-04-22T10:05:13.446 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c4bb947cd4bb4432a0696d254cef59fe Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 693, in cluster mons = get_mons( File "/home/teuthworker/src/git.ceph.com_ceph_226cba1c5219f5bb8d0cd056721711c87d9bf2fd/qa/tasks/ceph.py", line 510, in get_mons assert mons AssertionError 2024-04-22T10:05:13.450 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T10:05:13.457 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T10:05:13.508 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T10:05:13.509 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T10:05:13.533 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T10:05:13.533 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T10:05:13.558 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T10:05:13.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T10:05:13.583 INFO:teuthology.task.internal:Duration was 0.541492 seconds 2024-04-22T10:05:13.583 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T10:05:13.608 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T10:05:13.609 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T10:05:13.661 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} duration: 0.5414917469024658 failure_reason: '' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c4bb947cd4bb4432a0696d254cef59fe status: fail success: false 2024-04-22T10:05:13.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T10:05:13.779 INFO:teuthology.run:FAIL