2024-04-26T19:05:37.774 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d
2024-04-26T19:05:37.775 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7674997 --verbose --description rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} --name yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7674997/orig.config.yaml
2024-04-26T19:05:37.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-26T19:05:37.918 INFO:teuthology.run:Config:
  arch: x86_64
  archive_path: /home/teuthworker/archive/yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7674997
  branch: wip-yuri3-testing-2024-04-05-0825
  description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic
    msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery}
    supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}
  email: null
  first_in_suite: false
  job_id: '7674997'
  kernel:
    kdb: 1
    sha1: distro
  ktype: distro
  last_in_suite: false
  machine_type: smithi
  name: yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  os_type: ubuntu
  os_version: '22.04'
  overrides:
    admin_socket:
      branch: wip-yuri3-testing-2024-04-05-0825
    ceph:
      conf:
        global:
          mon client directed command retry: 5
          mon election default strategy: 1
          ms inject socket failures: 2500
          ms tcp read timeout: 5
          osd_async_recovery_min_cost: 1
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon min osdmap epochs: 50
          mon osdmap full prune interval: 2
          mon osdmap full prune min: 15
          mon osdmap full prune txsize: 2
          mon scrub interval: 300
          paxos service trim min: 10
        osd:
          bluestore block size: 96636764160
          bluestore compression algorithm: zstd
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          bluestore zero block detection: true
          debug bluefs: 20
          debug bluestore: 20
          debug ms: 1
          debug osd: 20
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd blocked scrub grace period: 3600
          osd debug reject backfill probability: 0.3
          osd debug verify cached snaps: true
          osd debug verify missing on start: true
          osd delete sleep: 1
          osd failsafe full ratio: 0.95
          osd max backfills: 3
          osd max markdown count: 1000
          osd mclock override recovery settings: true
          osd mclock profile: high_recovery_ops
          osd objectstore: bluestore
          osd op queue: debug_random
          osd op queue cut off: debug_random
          osd scrub max interval: 120
          osd scrub min interval: 60
          osd snap trim sleep: 2
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(OSD_SLOW_PING_TIME
      - but it is still running
      - objects unfound and apparently lost
      - overall HEALTH_
      - \(OSDMAP_FLAGS\)
      - \(OSD_
      - \(PG_
      - \(POOL_
      - \(CACHE_POOL_
      - \(SMALLER_PGP_NUM\)
      - \(OBJECT_
      - \(SLOW_OPS\)
      - \(REQUEST_SLOW\)
      - \(TOO_FEW_PGS\)
      - slow request
      - timeout on replica
      - late reservation from
      - MON_DOWN
      - OSDMAP_FLAGS
      - OSD_DOWN
      - PG_DEGRADED
      - PG_AVAILABILITY
      - POOL_APP_NOT_ENABLED
      - mons down
      - mon down
      - out of quorum
      sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
    roles:
    - - mon.a
      - mgr.y
      - osd.0
      - osd.4
      - osd.8
      - osd.12
    - - mon.b
      - osd.1
      - osd.5
      - osd.9
      - osd.13
    - - mon.c
      - osd.2
      - osd.6
      - osd.10
      - osd.14
    - - mgr.x
      - osd.3
      - osd.7
      - osd.11
      - osd.15
      - client.0
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-yuri3-testing-2024-04-05-0825
      sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
  owner: scheduled_yuriw@teuthology
  priority: 99
  repo: https://github.com/ceph/ceph-ci.git
  seed: 3753
  sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
  sleep_before_teardown: 0
  subset: 111/120000
  suite: rados
  suite_branch: wip-yuri3-testing-2024-04-05-0825
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci.git
  suite_sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
  tasks:
  - install: null
  - ceph:
      conf:
        osd:
          debug monc: 20
  - thrashosds:
      bluestore_new_sharding: random
      chance_bluestore_reshard: 1
      chance_pgnum_grow: 1
      chance_pgnum_shrink: 1
      chance_pgpnum_fix: 1
      min_in: 2
      thrash_hosts: true
      timeout: 1200
  - rados:
      clients:
      - client.0
      ec_pool: true
      erasure_code_crush:
        id: 86
        max_size: 6
        min_size: 3
        name: jerasure86crush
        steps:
        - set_chooseleaf_tries 5
        - set_choose_tries 100
        - take default class hdd
        - choose indep 4 type host
        - choose indep 4 type osd
        - emit
        type: erasure
      erasure_code_profile:
        crush-failure-domain: osd
        k: 8
        m: 6
        name: jerasure86profile
        plugin: jerasure
        technique: reed_sol_van
      max_in_flight: 64
      max_seconds: 600
      objects: 1024
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 400000
      size: 16384
      write_append_excl: false
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17
  timestamp: 2024-04-26_18:18:24
  tube: smithi
  user: yuriw
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885
2024-04-26T19:05:37.918 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa; will attempt to use it
2024-04-26T19:05:37.919 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks
2024-04-26T19:05:37.919 INFO:teuthology.run_tasks:Running task internal.check_packages...
2024-04-26T19:05:37.920 INFO:teuthology.task.internal:Checking packages...
2024-04-26T19:05:37.945 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5d349943c59c9485df060d6adb0594f3940ec0eb'
2024-04-26T19:05:37.945 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2024-04-26T19:05:37.945 INFO:teuthology.packaging:ref: None
2024-04-26T19:05:37.945 INFO:teuthology.packaging:tag: None
2024-04-26T19:05:37.945 INFO:teuthology.packaging:branch: wip-yuri3-testing-2024-04-05-0825
2024-04-26T19:05:37.945 INFO:teuthology.packaging:sha1: 5d349943c59c9485df060d6adb0594f3940ec0eb
2024-04-26T19:05:37.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2024-04-05-0825
2024-04-26T19:05:38.085 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3211-g5d349943-1jammy
2024-04-26T19:05:38.086 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2024-04-26T19:05:38.094 INFO:teuthology.task.internal:no buildpackages task found
2024-04-26T19:05:38.094 INFO:teuthology.run_tasks:Running task internal.save_config...
2024-04-26T19:05:38.109 INFO:teuthology.task.internal:Saving configuration
2024-04-26T19:05:38.123 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2024-04-26T19:05:38.130 INFO:teuthology.run_tasks:Running task kernel...
2024-04-26T19:05:38.146 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'}
2024-04-26T19:05:38.147 INFO:teuthology.task.kernel:config {}, timeout 300
2024-04-26T19:05:38.147 INFO:teuthology.run_tasks:Running task internal.archive_upload...
2024-04-26T19:05:38.156 INFO:teuthology.run_tasks:Running task internal.timer...
2024-04-26T19:05:38.171 INFO:teuthology.task.internal:Starting timer...
2024-04-26T19:05:38.171 INFO:teuthology.run_tasks:Running task ansible.cephlab...
2024-04-26T19:05:38.186 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main
2024-04-26T19:05:38.206 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}]
2024-04-26T19:05:38.206 INFO:teuthology.task.ansible:There are no remotes; skipping playbook execution
2024-04-26T19:05:38.206 INFO:teuthology.run_tasks:Running task clock...
2024-04-26T19:05:38.217 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew...
2024-04-26T19:05:38.217 INFO:teuthology.run_tasks:Running task install...
2024-04-26T19:05:38.232 DEBUG:teuthology.task.install:project ceph
2024-04-26T19:05:38.232 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}
2024-04-26T19:05:38.232 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}
2024-04-26T19:05:38.232 INFO:teuthology.task.install:Using flavor: default
2024-04-26T19:05:38.238 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']}
2024-04-26T19:05:38.238 INFO:teuthology.task.install:extra packages: []
2024-04-26T19:05:38.238 INFO:teuthology.task.install.util:Shipping valgrind.supp...
2024-04-26T19:05:38.238 INFO:teuthology.task.install.util:Shipping 'daemon-helper'...
2024-04-26T19:05:38.239 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'...
2024-04-26T19:05:38.239 INFO:teuthology.task.install.util:Shipping 'stdin-killer'...
2024-04-26T19:05:38.239 INFO:teuthology.run_tasks:Running task ceph...
2024-04-26T19:05:38.326 INFO:tasks.ceph:Making ceph log dir writeable by non-root...
2024-04-26T19:05:38.326 INFO:tasks.ceph:Disabling ceph logrotate...
2024-04-26T19:05:38.326 INFO:tasks.ceph:Creating extra log directories...
2024-04-26T19:05:38.331 INFO:tasks.ceph:Creating ceph cluster ceph...
2024-04-26T19:05:38.331 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True}
2024-04-26T19:05:38.331 INFO:tasks.ceph:ctx.config {'arch': 'x86_64', 'archive_path': '/home/teuthworker/archive/yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/7674997', 'branch': 'wip-yuri3-testing-2024-04-05-0825', 'description': 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}', 'email': None, 'first_in_suite': False, 'job_id': '7674997', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-26_18:18:24-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2024-04-05-0825'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2024-04-05-0825', 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'seed': 3753, 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri3-testing-2024-04-05-0825', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.add_remotes': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.archive_upload': None}, {'internal.timer': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '5d349943c59c9485df060d6adb0594f3940ec0eb', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_crush': {'id': 86, 'max_size': 6, 'min_size': 3, 'name': 'jerasure86crush', 'steps': ['set_chooseleaf_tries 5', 'set_choose_tries 100', 'take default class hdd', 'choose indep 4 type host', 'choose indep 4 type osd', 'emit'], 'type': 'erasure'}, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 8, 'm': 6, 'name': 'jerasure86profile', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'max_in_flight': 64, 'max_seconds': 600, 'objects': 1024, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400000, 'size': 16384, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_18:18:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}
2024-04-26T19:05:38.332 INFO:tasks.ceph:remote_to_roles_to_devs: {}
2024-04-26T19:05:38.332 INFO:tasks.ceph:Generating config...
2024-04-26T19:05:38.332 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-26T19:05:38.333 INFO:tasks.ceph:Archiving crash dumps...
2024-04-26T19:05:38.334 INFO:tasks.ceph:Compressing logs...
2024-04-26T19:05:38.334 INFO:tasks.ceph:Archiving logs...
2024-04-26T19:05:38.334 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 1913, in task
    with contextutil.nested(*subtasks):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 252, in ceph_log
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-26T19:05:38.537 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5571f2578e674c42bbb74b7e057fc02e
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 1913, in task
    with contextutil.nested(*subtasks):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 252, in ceph_log
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/github.com_ceph_ceph-c_5d349943c59c9485df060d6adb0594f3940ec0eb/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-26T19:05:38.541 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2024-04-26T19:05:38.554 DEBUG:teuthology.run_tasks:Unwinding manager install
2024-04-26T19:05:38.568 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer...
2024-04-26T19:05:38.569 DEBUG:teuthology.run_tasks:Unwinding manager clock
2024-04-26T19:05:38.583 INFO:teuthology.task.clock:Checking final clock skew...
2024-04-26T19:05:38.583 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-04-26T19:05:38.597 INFO:teuthology.task.ansible:Skipping ansible cleanup...
2024-04-26T19:05:38.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer
2024-04-26T19:05:38.612 INFO:teuthology.task.internal:Duration was 0.440874 seconds
2024-04-26T19:05:38.612 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2024-04-26T19:05:38.626 INFO:teuthology.task.internal:Not uploading archives.
2024-04-26T19:05:38.626 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-04-26T19:05:38.659 INFO:teuthology.run:Summary data:
description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic
  msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery}
  supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}
duration: 0.4408740997314453
failure_reason: ''
flavor: default
owner: scheduled_yuriw@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5571f2578e674c42bbb74b7e057fc02e
status: fail
success: false

2024-04-26T19:05:38.660 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-26T19:05:38.737 INFO:teuthology.run:FAIL