2024-04-12T16:37:55.356 INFO:root:teuthology version: 0.0.1.dev267+g6c63784
2024-04-12T16:37:55.356 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} --name rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654180 --verbose -- /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654180/orig.config.yaml
2024-04-12T16:37:55.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-12T16:37:55.489 INFO:teuthology.run:Config:
  arch: x86_64
  archive_path: /home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654180
  branch: wip-yuri10-testing-2024-03-24-1159
  description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic
    msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery}
    supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}
  email: null
  first_in_suite: false
  job_id: '7654180'
  kernel:
    kdb: 1
    sha1: distro
  ktype: distro
  last_in_suite: false
  machine_type: smithi
  name: rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  os_type: ubuntu
  os_version: '22.04'
  overrides:
    admin_socket:
      branch: wip-yuri10-testing-2024-03-24-1159
    ceph:
      conf:
        global:
          mon client directed command retry: 5
          mon election default strategy: 1
          ms inject socket failures: 5000
          osd_async_recovery_min_cost: 1
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon min osdmap epochs: 50
          mon osdmap full prune interval: 2
          mon osdmap full prune min: 15
          mon osdmap full prune txsize: 2
          mon scrub interval: 300
          paxos service trim min: 10
        osd:
          bluestore block size: 96636764160
          bluestore compression algorithm: snappy
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          bluestore zero block detection: true
          debug bluefs: 20
          debug bluestore: 20
          debug ms: 1
          debug osd: 20
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd beacon report interval: 30
          osd blocked scrub grace period: 3600
          osd debug verify cached snaps: true
          osd debug verify missing on start: true
          osd failsafe full ratio: 0.95
          osd heartbeat use min delay socket: true
          osd map cache size: 1
          osd max backfills: 6
          osd max markdown count: 1000
          osd mclock override recovery settings: true
          osd mclock profile: high_recovery_ops
          osd objectstore: bluestore
          osd op queue: debug_random
          osd op queue cut off: debug_random
          osd scrub during recovery: false
          osd scrub max interval: 120
          osd scrub min interval: 60
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(OSD_SLOW_PING_TIME
      - but it is still running
      - objects unfound and apparently lost
      - osd_map_cache_size
      - overall HEALTH_
      - \(OSDMAP_FLAGS\)
      - \(OSD_
      - \(PG_
      - \(POOL_
      - \(CACHE_POOL_
      - \(SMALLER_PGP_NUM\)
      - \(OBJECT_
      - \(SLOW_OPS\)
      - \(REQUEST_SLOW\)
      - \(TOO_FEW_PGS\)
      - slow request
      - timeout on replica
      - late reservation from
      - MON_DOWN
      - OSDMAP_FLAGS
      - OSD_DOWN
      - PG_DEGRADED
      - PG_AVAILABILITY
      - POOL_APP_NOT_ENABLED
      - mons down
      - mon down
      - out of quorum
      sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
    roles:
    - - mon.a
      - mgr.y
      - osd.0
      - osd.4
      - osd.8
      - osd.12
    - - mon.b
      - osd.1
      - osd.5
      - osd.9
      - osd.13
    - - mon.c
      - osd.2
      - osd.6
      - osd.10
      - osd.14
    - - mgr.x
      - osd.3
      - osd.7
      - osd.11
      - osd.15
      - client.0
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-yuri10-testing-2024-03-24-1159
      sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
  owner: scheduled_rzarzynski@teuthology
  priority: 85
  repo: https://git.ceph.com/ceph-ci.git
  seed: 8148
  sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
  sleep_before_teardown: 0
  subset: 111/120000
  suite: rados
  suite_branch: wip-yuri10-testing-2024-03-24-1159
  suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa
  suite_relpath: qa
  suite_repo: https://git.ceph.com/ceph-ci.git
  suite_sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
  tasks:
  - install: null
  - ceph:
      conf:
        osd:
          debug monc: 20
  - thrashosds:
      chance_pgnum_grow: 0.25
      chance_pgnum_shrink: 0.25
      chance_pgpnum_fix: 0.25
      chance_test_map_discontinuity: 2
      map_discontinuity_sleep_time: 200
      min_in: 2
      timeout: 1800
  - rados:
      clients:
      - client.0
      ec_pool: true
      erasure_code_crush:
        id: 86
        max_size: 6
        min_size: 3
        name: jerasure86crush
        steps:
        - set_chooseleaf_tries 5
        - set_choose_tries 100
        - take default class hdd
        - choose indep 4 type host
        - choose indep 4 type osd
        - emit
        type: erasure
      erasure_code_profile:
        crush-failure-domain: osd
        k: 8
        m: 6
        name: jerasure86profile
        plugin: jerasure
        technique: reed_sol_van
      max_in_flight: 64
      max_seconds: 600
      objects: 1024
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 400000
      size: 16384
      write_append_excl: false
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab
  timestamp: 2024-04-12_15:15:07
  tube: smithi
  user: rzarzynski
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885
2024-04-12T16:37:55.489 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa; will attempt to use it
2024-04-12T16:37:55.489 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks
2024-04-12T16:37:55.490 INFO:teuthology.run_tasks:Running task internal.check_packages...
2024-04-12T16:37:55.491 INFO:teuthology.task.internal:Checking packages...
2024-04-12T16:37:55.515 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'
2024-04-12T16:37:55.515 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2024-04-12T16:37:55.516 INFO:teuthology.packaging:ref: None
2024-04-12T16:37:55.516 INFO:teuthology.packaging:tag: None
2024-04-12T16:37:55.516 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-24-1159
2024-04-12T16:37:55.516 INFO:teuthology.packaging:sha1: bf9c5618cc018927fa77780d25b6deb1f5dc254d
2024-04-12T16:37:55.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri10-testing-2024-03-24-1159
2024-04-12T16:37:55.732 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2847-gbf9c5618-1jammy
2024-04-12T16:37:55.734 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2024-04-12T16:37:55.741 INFO:teuthology.task.internal:no buildpackages task found
2024-04-12T16:37:55.741 INFO:teuthology.run_tasks:Running task internal.save_config...
2024-04-12T16:37:55.751 INFO:teuthology.task.internal:Saving configuration
2024-04-12T16:37:55.765 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2024-04-12T16:37:55.771 INFO:teuthology.run_tasks:Running task kernel...
2024-04-12T16:37:55.789 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'}
2024-04-12T16:37:55.789 INFO:teuthology.task.kernel:config {}, timeout 300
2024-04-12T16:37:55.789 INFO:teuthology.run_tasks:Running task internal.archive_upload...
2024-04-12T16:37:55.795 INFO:teuthology.run_tasks:Running task internal.timer...
2024-04-12T16:37:55.805 INFO:teuthology.task.internal:Starting timer...
2024-04-12T16:37:55.806 INFO:teuthology.run_tasks:Running task ansible.cephlab...
2024-04-12T16:37:55.817 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main
2024-04-12T16:37:55.833 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}]
2024-04-12T16:37:55.834 INFO:teuthology.task.ansible:There are no remotes; skipping playbook execution
2024-04-12T16:37:55.834 INFO:teuthology.run_tasks:Running task clock...
2024-04-12T16:37:55.841 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew...
2024-04-12T16:37:55.842 INFO:teuthology.run_tasks:Running task install...
2024-04-12T16:37:55.849 DEBUG:teuthology.task.install:project ceph
2024-04-12T16:37:55.849 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}}
2024-04-12T16:37:55.849 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}
2024-04-12T16:37:55.849 INFO:teuthology.task.install:Using flavor: default
2024-04-12T16:37:55.855 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']}
2024-04-12T16:37:55.856 INFO:teuthology.task.install:extra packages: []
2024-04-12T16:37:55.856 INFO:teuthology.task.install.util:Shipping valgrind.supp...
2024-04-12T16:37:55.856 INFO:teuthology.task.install.util:Shipping 'daemon-helper'...
2024-04-12T16:37:55.856 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'...
2024-04-12T16:37:55.856 INFO:teuthology.task.install.util:Shipping 'stdin-killer'...
2024-04-12T16:37:55.857 INFO:teuthology.run_tasks:Running task ceph...
2024-04-12T16:37:55.941 INFO:tasks.ceph:Making ceph log dir writeable by non-root...
2024-04-12T16:37:55.941 INFO:tasks.ceph:Disabling ceph logrotate...
2024-04-12T16:37:55.941 INFO:tasks.ceph:Creating extra log directories...
2024-04-12T16:37:55.946 INFO:tasks.ceph:Creating ceph cluster ceph...
2024-04-12T16:37:55.946 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True}
2024-04-12T16:37:55.946 INFO:tasks.ceph:ctx.config {'arch': 'x86_64', 'archive_path': '/home/teuthworker/archive/rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7654180', 'branch': 'wip-yuri10-testing-2024-03-24-1159', 'description': 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}', 'email': None, 'first_in_suite': False, 'job_id': '7654180', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rzarzynski-2024-04-12_15:15:07-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2024-03-24-1159'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}}, 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2024-03-24-1159', 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d'}}, 'owner': 'scheduled_rzarzynski@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'seed': 8148, 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2024-03-24-1159', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.add_remotes': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.archive_upload': None}, {'internal.timer': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'osd_async_recovery_min_cost': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'bf9c5618cc018927fa77780d25b6deb1f5dc254d', 'cluster': 'ceph'}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_crush': {'id': 86, 'max_size': 6, 'min_size': 3, 'name': 'jerasure86crush', 'steps': ['set_chooseleaf_tries 5', 'set_choose_tries 100', 'take default class hdd', 'choose indep 4 type host', 'choose indep 4 type osd', 'emit'], 'type': 'erasure'}, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 8, 'm': 6, 'name': 'jerasure86profile', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'max_in_flight': 64, 'max_seconds': 600, 'objects': 1024, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400000, 'size': 16384, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-12_15:15:07', 'tube': 'smithi', 'user': 'rzarzynski', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'}
2024-04-12T16:37:55.947 INFO:tasks.ceph:remote_to_roles_to_devs: {}
2024-04-12T16:37:55.947 INFO:tasks.ceph:Generating config...
2024-04-12T16:37:55.947 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-12T16:37:55.948 INFO:tasks.ceph:Archiving crash dumps...
2024-04-12T16:37:55.948 INFO:tasks.ceph:Compressing logs...
2024-04-12T16:37:55.949 INFO:tasks.ceph:Archiving logs...
2024-04-12T16:37:55.949 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 1913, in task
    with contextutil.nested(*subtasks):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 252, in ceph_log
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-12T16:37:56.157 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3b503565af0c485d8b9c92203c5727a3
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 1913, in task
    with contextutil.nested(*subtasks):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 252, in ceph_log
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 693, in cluster
    mons = get_mons(
  File "/home/teuthworker/src/git.ceph.com_ceph-c_bf9c5618cc018927fa77780d25b6deb1f5dc254d/qa/tasks/ceph.py", line 510, in get_mons
    assert mons
AssertionError
2024-04-12T16:37:56.161 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2024-04-12T16:37:56.168 DEBUG:teuthology.run_tasks:Unwinding manager install
2024-04-12T16:37:56.182 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer...
2024-04-12T16:37:56.183 DEBUG:teuthology.run_tasks:Unwinding manager clock
2024-04-12T16:37:56.192 INFO:teuthology.task.clock:Checking final clock skew...
2024-04-12T16:37:56.193 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-04-12T16:37:56.203 INFO:teuthology.task.ansible:Skipping ansible cleanup...
2024-04-12T16:37:56.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer
2024-04-12T16:37:56.211 INFO:teuthology.task.internal:Duration was 0.405720 seconds
2024-04-12T16:37:56.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2024-04-12T16:37:56.218 INFO:teuthology.task.internal:Not uploading archives.
2024-04-12T16:37:56.218 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-04-12T16:37:56.240 INFO:teuthology.run:Summary data:
description: rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic
  msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery}
  supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}
duration: 0.4057197570800781
failure_reason: ''
flavor: default
owner: scheduled_rzarzynski@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3b503565af0c485d8b9c92203c5727a3
status: fail
success: false

2024-04-12T16:37:56.240 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-12T16:37:56.325 INFO:teuthology.run:FAIL