2023-07-27T05:36:58.828 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-27T05:36:58.829 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi --verbose --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --archive /home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595 -- /home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595/orig.config.yaml
2023-07-27T05:36:58.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-27T05:36:58.969 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595
  branch: wip-yuri6-testing-2023-07-24-0819-pacific
  description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic
    msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery}
    supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
  email: yweinste@redhat.com
  first_in_suite: false
  job_id: '7353595'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  name: yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  openstack:
  - volumes:
      count: 4
      size: 10
  os_type: rhel
  os_version: '8.4'
  overrides:
    admin_socket:
      branch: wip-yuri6-testing-2023-07-24-0819-pacific
    ceph:
      conf:
        global:
          mon election default strategy: 1
          osd debug inject dispatch delay duration: 0.1
          osd debug inject dispatch delay probability: 0.1
          osd_object_clean_region_max_num_intervals: 1000
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon scrub interval: 300
        osd:
          bluestore block size: 96636764160
          bluestore compression algorithm: zstd
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          debug bluefs: 20
          debug bluestore: 20
          debug ms: 1
          debug osd: 20
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd debug reject backfill probability: 0.3
          osd debug verify cached snaps: true
          osd debug verify missing on start: true
          osd failsafe full ratio: 0.95
          osd max backfills: 3
          osd max markdown count: 1000
          osd objectstore: bluestore
          osd op queue: debug_random
          osd op queue cut off: debug_random
          osd scrub max interval: 120
          osd scrub min interval: 60
          osd shutdown pgref assert: true
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - but it is still running
      - objects unfound and apparently lost
      - slow request
      - overall HEALTH_
      - \(OSDMAP_FLAGS\)
      - \(OSD_
      - \(PG_
      - \(POOL_
      - \(CACHE_POOL_
      - \(SMALLER_PGP_NUM\)
      - \(OBJECT_
      - \(SLOW_OPS\)
      - \(REQUEST_SLOW\)
      - \(TOO_FEW_PGS\)
      - slow request
      sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
    install:
      ceph:
        flavor: default
        sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
    selinux:
      whitelist:
      - scontext=system_u:system_r:logrotate_t:s0
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-yuri6-testing-2023-07-24-0819-pacific
      sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
  owner: scheduled_yuriw@teuthology
  priority: 95
  repo: https://github.com/ceph/ceph-ci.git
  roles:
  - - mon.a
    - mgr.y
    - osd.0
    - osd.4
    - osd.8
    - osd.12
  - - mon.b
    - osd.1
    - osd.5
    - osd.9
    - osd.13
  - - mon.c
    - osd.2
    - osd.6
    - osd.10
    - osd.14
  - - mgr.x
    - osd.3
    - osd.7
    - osd.11
    - osd.15
    - client.0
  seed: 6824
  sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
  sleep_before_teardown: 0
  subset: 111/120000
  suite: rados
  suite_branch: wip-yuri6-testing-2023-07-24-0819-pacific
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_2dfc9d0e12cf19dcb739a527dae47ac94e230bb4/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci.git
  suite_sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
  targets:
    smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCidOQdcNgBcf+iMVZipIK9uHHb7n2xkcAe+426Jljyced+j+7WYYoSHmEnD9cdDt2/H3GJy2bpprd/PeyVcTsJ59ITT+jfoTqgBT3hndd0CJV3TfbiMdYkb4X7k6KYO4uRvR/NbBGKGGF7T1HYoMvZLhQSi+9NFn2P+eqHbrnz4OKExtiCLaO0ENNqt8KCrl6dL99vi4S1FxVE34/0EE1Fqbvx/U+ZnwomjkuF+MqkeyhILpe6kUrRSWphJDzGAcodN+41Y9T1zLyzknYBenkPaedAP4mJL6Y7r+gs/kE+4jFP6TXEWeKP075MqeiLt4PTFNhGL/Hjkm+BO098aMayPBGd1wDsmpF8cQER9HiV1NLRJdoswNaVb/kDhiAYsW+i70jv3cvG/YOrBnCaTR8xxSDJBlKaZsH+66SwX5amdRoYv7DbI8FtqPCKfQ4JE5KC79VFcuPoQBcqPXyoiCzdvu9DZq/8IZWXHZBsw1WrjjFFLPyNQcr16aDUQLs1cgs=
    smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0B6MBdPyQDyIerGqN2T867JvK/FN5o0tDAPwRSrkQiRLFJb+FzBmjg2a/EA7Vf9HXnr3Y6cCmFwPYL73Nk+0EkoFizYj2jDbl5ATwV4e/9nPoz6X18ERHabHoRmg9LxCXtmp1ANzj6W4JP7x1+eJgi8LVMvtMwBGICaI68tBgHe4EgzkkxlnS5CTLWhg5lzFOq8K5wQcvj3JPMIoOxXYaD/93t1XNWRb6Fimbcew4+5vDRug5nh+WPxfZ6TrEMg5Bx2TOY5AFXNeQcyrsmw7P/w55GckstBflnRB35MwIAdTZEyxBsPFrKTZKHSlSblHcQWk3SbkpqAVENpSnh6gJrCS05HvBgBqEl9iOygvGvJXL/1D41GVj4k2k89gddoDzf2C3AHv9WyFTW9KjLTJoe+hA6JtICbp7L7D+AIJ7ZToUkCu3k8CJQU19T6ebLF+hciP/a93am4TzOXvtTUMM1EOKIKsob38xx174OeC43ZdXY4i6CIJi8MAPKPYpWJM=
    smithi117.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbcCPGgUjfxc73zqpDLtp5Tgw2SJVyMibbOTg9/DWcH/ulSDNjQEnIaIRES19+mQHH9pmSdlQ9hbbeu7XoWcNgQL511/1pJ+heZIZpmqYMMWxUNPABDUwzG1KfX6YBIV6kXIDqWZuU2FPe0lrwBkYnKig74kmcXXhz6BhIKewBvTnFMSx2hvQZB0imkbCFPePlhovyLWPbGQ0eRcxrK3Fjd0UJnks/p7c0AfyfT4IX9u6R9XHiaBKkHjBze/V3Ur09lfFZ+xbt8BP/x4hRdWDYphd+q89F7cR9etnozlVhJ2cua8xAA2Z7v0zwQr8HSqvwuTjuw7imJjhCoOtishLyOd/nEVWrc+7ddYpb6jcfVwKdz1GYH5in6Vp85+OelWZgBPJNdBi0fkth/2HsSsZJAggqpTcxLXB9eWT4kMGKQMCkmdCZ6PWIy59xm1Ixac//qJlIwfcEi1bzanLoM84LuKYzqbZQPxoZMBr0GxhyTJ+IPJCllyp2Q8KB+B96lvc=
    smithi173.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHuEyMAcKbX0BeS6qr5UIHfO4ebNnNAavfyjueY5OHZL6DLC3g5Pu/jvbLatm14S5fSh21/13JUc1XRRFX/5GI9WayDJUFkWUTfnXozkLz6m/uF1oq3OexqSR3HOJxC/Sc/fIkqRvWoTwqZjQbnkLTYtPf0M8nNdiqyYzSkA+cIdqyDlU5iJcL9Iv6GeGNgWklAMsX7ztFFcC9gyZxT33w5TUDbBbaqOGiGaxPU2XCCpOGoCIdstU6HaBCiRaeccJhmEAa61fy15CEBhwxe4AsSkjDnawxA+6lhUNkZY2JWXJFANAPii/iDwcuPMwedY0V63Ewd2jsHppGufM5Aa3Hb4mIXBISo9Gu6Kw5iwuQTPGAvUF7f5gSR4asiKFrvVH7Yjfkrr4EYY70H9S0VUvlagJXw7wz0+epWOmbmiNGFDiLVnomqzO4tebAhBfq44VYzE+73flFxx4te7Y1JJ7yOygTBXaTHOsopYP9Tj/pjHM7zTxGu4uQB/VqCTkI/mU=
  tasks:
  - install: null
  - ceph:
      conf:
        osd:
          debug monc: 20
  - thrashosds:
      aggressive_pg_num_changes: false
      chance_pgnum_grow: 1
      chance_pgnum_shrink: 1
      chance_pgpnum_fix: 1
      min_in: 8
      timeout: 1200
  - rados:
      clients:
      - client.0
      ec_pool: true
      erasure_code_profile:
        c: 2
        crush-failure-domain: osd
        k: 4
        m: 3
        name: shecprofile
        plugin: shec
      objects: 50
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 400
      write_append_excl: false
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 407880c6d3fb77318fff01c863715090f9c2de69
  timestamp: 2023-07-26_15:54:22
  tube: smithi
  user: yuriw
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459
2023-07-27T05:36:58.970 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_2dfc9d0e12cf19dcb739a527dae47ac94e230bb4/qa; will attempt to use it
2023-07-27T05:36:58.970 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_2dfc9d0e12cf19dcb739a527dae47ac94e230bb4/qa/tasks
2023-07-27T05:36:58.971 INFO:teuthology.run_tasks:Running task internal.check_packages...
2023-07-27T05:36:58.972 INFO:teuthology.task.internal:Checking packages...
2023-07-27T05:36:59.006 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '2dfc9d0e12cf19dcb739a527dae47ac94e230bb4'
2023-07-27T05:36:59.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2023-07-27T05:36:59.007 INFO:teuthology.packaging:ref: None
2023-07-27T05:36:59.008 INFO:teuthology.packaging:tag: None
2023-07-27T05:36:59.008 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-07-24-0819-pacific
2023-07-27T05:36:59.008 INFO:teuthology.packaging:sha1: 2dfc9d0e12cf19dcb739a527dae47ac94e230bb4
2023-07-27T05:36:59.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-07-24-0819-pacific
2023-07-27T05:36:59.879 INFO:teuthology.task.internal:Found packages for ceph version 16.2.13-360.g2dfc9d0e
2023-07-27T05:36:59.879 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2023-07-27T05:36:59.885 INFO:teuthology.task.internal:no buildpackages task found
2023-07-27T05:36:59.886 INFO:teuthology.run_tasks:Running task internal.save_config...
2023-07-27T05:36:59.894 INFO:teuthology.task.internal:Saving configuration
2023-07-27T05:36:59.910 INFO:teuthology.run_tasks:Running task internal.check_lock...
2023-07-27T05:36:59.916 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-27T05:37:14.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:58:31-rados-wip-yuri8-testing-2023-07-24-0819-quincy-distro-default-smithi/7353596', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.917371', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsu/NginRCYGnhjQu9knC8+xlUuKpnWxMkQ8zvqsXOZJGfkc+R5dy6SmTVP5UAfSy6v4LwC1vUvlmm5ED9HJKs='}
2023-07-27T05:37:19.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.481313', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIzGXzTnZlJaWyqu6L7TZOC5gwgwvfsGIZQ9Hsl+Flg+5OymlPDtdmt3V74tHJvRH/qk08U7OyrmVu2R9Dm6oIU='}
2023-07-27T05:37:25.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.478195', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBODUVtknqj6z6RV1xVpQAedwh82eS+ypBHdT7CiXN3f2TAFk09kV2rUO32Uo1AwagbL/u0BlAHW66iE3lyhGRPA='}
2023-07-27T05:37:30.044 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.475095', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDDQrq9kCbZm7LkB1muf2DQQ3PMWkzbDYXYr24/MYCr/gWhSsotTs02/Oj7tNjZqO1K9pYD8ahyGKEuKyhevClM='}
2023-07-27T05:37:30.045 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2023-07-27T05:37:30.051 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12']
2023-07-27T05:37:30.052 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13']
2023-07-27T05:37:30.052 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14']
2023-07-27T05:37:30.053 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']
2023-07-27T05:37:30.053 INFO:teuthology.run_tasks:Running task console_log...
2023-07-27T05:37:40.120 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fa0bfd7fa30>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fa0bf044dc0>, signals=[15])
2023-07-27T05:37:40.121 INFO:teuthology.run_tasks:Running task internal.connect...
2023-07-27T05:37:40.127 INFO:teuthology.task.internal:Opening connections...
2023-07-27T05:37:40.128 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com
2023-07-27T05:37:40.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-27T05:37:40.199 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com
2023-07-27T05:37:40.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-27T05:37:40.271 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com
2023-07-27T05:37:40.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-27T05:37:40.341 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com
2023-07-27T05:37:40.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-27T05:37:40.413 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2023-07-27T05:37:40.419 DEBUG:teuthology.orchestra.run.smithi018:> uname -m
2023-07-27T05:37:40.434 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64
2023-07-27T05:37:40.435 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release
2023-07-27T05:37:40.488 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Red Hat Enterprise Linux"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="8.4 (Ootpa)"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:ID="rhel"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="fedora"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="8.4"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el8"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-07-27T05:37:40.489 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.redhat.com/"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-07-27T05:37:40.490 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-27T05:37:40.491 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-07-27T05:37:40.491 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server
2023-07-27T05:37:40.512 DEBUG:teuthology.orchestra.run.smithi067:> uname -m
2023-07-27T05:37:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64
2023-07-27T05:37:40.531 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release
2023-07-27T05:37:40.587 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Red Hat Enterprise Linux"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8.4 (Ootpa)"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:ID="rhel"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="fedora"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8.4"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8"
2023-07-27T05:37:40.588 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.redhat.com/"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:
2023-07-27T05:37:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-27T05:37:40.590 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-07-27T05:37:40.590 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-27T05:37:40.590 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-07-27T05:37:40.590 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server
2023-07-27T05:37:40.620 DEBUG:teuthology.orchestra.run.smithi117:> uname -m
2023-07-27T05:37:40.636 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64
2023-07-27T05:37:40.637 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release
2023-07-27T05:37:40.692 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="Red Hat Enterprise Linux"
2023-07-27T05:37:40.692 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="8.4 (Ootpa)"
2023-07-27T05:37:40.692 INFO:teuthology.orchestra.run.smithi117.stdout:ID="rhel"
2023-07-27T05:37:40.692 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="fedora"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="8.4"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el8"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://www.redhat.com/"
2023-07-27T05:37:40.693 INFO:teuthology.orchestra.run.smithi117.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-27T05:37:40.694 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-07-27T05:37:40.694 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server
2023-07-27T05:37:40.728 DEBUG:teuthology.orchestra.run.smithi173:> uname -m
2023-07-27T05:37:40.745 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64
2023-07-27T05:37:40.745 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release
2023-07-27T05:37:40.800 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="Red Hat Enterprise Linux"
2023-07-27T05:37:40.800 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8.4 (Ootpa)"
2023-07-27T05:37:40.800 INFO:teuthology.orchestra.run.smithi173.stdout:ID="rhel"
2023-07-27T05:37:40.800 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="fedora"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8.4"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://www.redhat.com/"
2023-07-27T05:37:40.801 INFO:teuthology.orchestra.run.smithi173.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-27T05:37:40.802 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-07-27T05:37:40.803 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2023-07-27T05:37:45.836 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2023-07-27T05:37:45.851 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2023-07-27T05:37:45.859 INFO:teuthology.task.internal:Checking for old test directory...
2023-07-27T05:37:45.860 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest
2023-07-27T05:37:45.862 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest
2023-07-27T05:37:45.864 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest
2023-07-27T05:37:45.866 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest
2023-07-27T05:37:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-07-27T05:37:45.874 ERROR:teuthology.task.internal:Host smithi018 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2023-07-27T05:37:45.883 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/run_tasks.py", line 84, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/task/internal/__init__.py", line 299, in check_conflict
    raise RuntimeError('Stale jobs detected, aborting.')
RuntimeError: Stale jobs detected, aborting.
2023-07-27T05:37:45.947 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=30e4f9c9a59343ed8b045bf18c4f0940
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/run_tasks.py", line 84, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/task/internal/__init__.py", line 299, in check_conflict
    raise RuntimeError('Stale jobs detected, aborting.')
RuntimeError: Stale jobs detected, aborting.
2023-07-27T05:37:45.948 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2023-07-27T05:37:45.964 INFO:teuthology.nuke:Checking targets against current locks
2023-07-27T05:37:51.054 INFO:teuthology.nuke:Not nuking smithi018.front.sepia.ceph.com because description doesn't match: yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi != /home/teuthworker/archive/yuriw-2023-07-26_15:58:31-rados-wip-yuri8-testing-2023-07-24-0819-quincy-distro-default-smithi/7353596
2023-07-27T05:37:56.084 DEBUG:teuthology.nuke:shortname: smithi067
2023-07-27T05:37:56.085 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-27T05:38:01.748 DEBUG:teuthology.nuke:shortname: smithi117
2023-07-27T05:38:01.749 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-27T05:38:01.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.481313', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0B6MBdPyQDyIerGqN2T867JvK/FN5o0tDAPwRSrkQiRLFJb+FzBmjg2a/EA7Vf9HXnr3Y6cCmFwPYL73Nk+0EkoFizYj2jDbl5ATwV4e/9nPoz6X18ERHabHoRmg9LxCXtmp1ANzj6W4JP7x1+eJgi8LVMvtMwBGICaI68tBgHe4EgzkkxlnS5CTLWhg5lzFOq8K5wQcvj3JPMIoOxXYaD/93t1XNWRb6Fimbcew4+5vDRug5nh+WPxfZ6TrEMg5Bx2TOY5AFXNeQcyrsmw7P/w55GckstBflnRB35MwIAdTZEyxBsPFrKTZKHSlSblHcQWk3SbkpqAVENpSnh6gJrCS05HvBgBqEl9iOygvGvJXL/1D41GVj4k2k89gddoDzf2C3AHv9WyFTW9KjLTJoe+hA6JtICbp7L7D+AIJ7ZToUkCu3k8CJQU19T6ebLF+hciP/a93am4TzOXvtTUMM1EOKIKsob38xx174OeC43ZdXY4i6CIJi8MAPKPYpWJM='}
2023-07-27T05:38:01.783 DEBUG:teuthology.nuke:shortname: smithi173
2023-07-27T05:38:01.783 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-27T05:38:01.807 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.478195', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbcCPGgUjfxc73zqpDLtp5Tgw2SJVyMibbOTg9/DWcH/ulSDNjQEnIaIRES19+mQHH9pmSdlQ9hbbeu7XoWcNgQL511/1pJ+heZIZpmqYMMWxUNPABDUwzG1KfX6YBIV6kXIDqWZuU2FPe0lrwBkYnKig74kmcXXhz6BhIKewBvTnFMSx2hvQZB0imkbCFPePlhovyLWPbGQ0eRcxrK3Fjd0UJnks/p7c0AfyfT4IX9u6R9XHiaBKkHjBze/V3Ur09lfFZ+xbt8BP/x4hRdWDYphd+q89F7cR9etnozlVhJ2cua8xAA2Z7v0zwQr8HSqvwuTjuw7imJjhCoOtishLyOd/nEVWrc+7ddYpb6jcfVwKdz1GYH5in6Vp85+OelWZgBPJNdBi0fkth/2HsSsZJAggqpTcxLXB9eWT4kMGKQMCkmdCZ6PWIy59xm1Ixac//qJlIwfcEi1bzanLoM84LuKYzqbZQPxoZMBr0GxhyTJ+IPJCllyp2Q8KB+B96lvc='}
2023-07-27T05:38:01.839 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 05:29:01.475095', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHuEyMAcKbX0BeS6qr5UIHfO4ebNnNAavfyjueY5OHZL6DLC3g5Pu/jvbLatm14S5fSh21/13JUc1XRRFX/5GI9WayDJUFkWUTfnXozkLz6m/uF1oq3OexqSR3HOJxC/Sc/fIkqRvWoTwqZjQbnkLTYtPf0M8nNdiqyYzSkA+cIdqyDlU5iJcL9Iv6GeGNgWklAMsX7ztFFcC9gyZxT33w5TUDbBbaqOGiGaxPU2XCCpOGoCIdstU6HaBCiRaeccJhmEAa61fy15CEBhwxe4AsSkjDnawxA+6lhUNkZY2JWXJFANAPii/iDwcuPMwedY0V63Ewd2jsHppGufM5Aa3Hb4mIXBISo9Gu6Kw5iwuQTPGAvUF7f5gSR4asiKFrvVH7Yjfkrr4EYY70H9S0VUvlagJXw7wz0+epWOmbmiNGFDiLVnomqzO4tebAhBfq44VYzE+73flFxx4te7Y1JJ7yOygTBXaTHOsopYP9Tj/pjHM7zTxGu4uQB/VqCTkI/mU='}
2023-07-27T05:38:01.854 INFO:teuthology.orchestra.console:Power off smithi067
2023-07-27T05:38:01.855 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-27T05:38:01.873 INFO:teuthology.orchestra.console:Power off smithi117
2023-07-27T05:38:01.874 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-27T05:38:01.889 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-27T05:38:01.889 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:01.901 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-27T05:38:01.901 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:01.911 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:01.925 INFO:teuthology.orchestra.console:Power off smithi173
2023-07-27T05:38:01.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-27T05:38:01.936 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:01.950 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-27T05:38:01.950 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:01.976 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:05.913 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:05.938 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:05.978 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:06.026 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:06.052 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:06.090 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:10.028 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:10.053 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:10.091 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:10.140 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-27T05:38:10.164 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-27T05:38:10.201 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-27T05:38:10.241 INFO:teuthology.orchestra.console:Power off for smithi067 completed
2023-07-27T05:38:10.265 INFO:teuthology.orchestra.console:Power off for smithi117 completed
2023-07-27T05:38:14.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:15.313 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-27T05:38:19.314 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-27T05:38:19.425 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-27T05:38:19.526 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2023-07-27T05:38:19.628 ERROR:teuthology.nuke:Could not nuke the following targets:
targets:
  smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCidOQdcNgBcf+iMVZipIK9uHHb7n2xkcAe+426Jljyced+j+7WYYoSHmEnD9cdDt2/H3GJy2bpprd/PeyVcTsJ59ITT+jfoTqgBT3hndd0CJV3TfbiMdYkb4X7k6KYO4uRvR/NbBGKGGF7T1HYoMvZLhQSi+9NFn2P+eqHbrnz4OKExtiCLaO0ENNqt8KCrl6dL99vi4S1FxVE34/0EE1Fqbvx/U+ZnwomjkuF+MqkeyhILpe6kUrRSWphJDzGAcodN+41Y9T1zLyzknYBenkPaedAP4mJL6Y7r+gs/kE+4jFP6TXEWeKP075MqeiLt4PTFNhGL/Hjkm+BO098aMayPBGd1wDsmpF8cQER9HiV1NLRJdoswNaVb/kDhiAYsW+i70jv3cvG/YOrBnCaTR8xxSDJBlKaZsH+66SwX5amdRoYv7DbI8FtqPCKfQ4JE5KC79VFcuPoQBcqPXyoiCzdvu9DZq/8IZWXHZBsw1WrjjFFLPyNQcr16aDUQLs1cgs=
2023-07-27T05:38:19.634 INFO:teuthology.run:Summary data:
description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic
  msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery}
  supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
failure_reason: Stale jobs detected, aborting.
owner: scheduled_yuriw@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=30e4f9c9a59343ed8b045bf18c4f0940
status: fail
success: false

2023-07-27T05:38:19.634 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-27T05:38:21.088 INFO:teuthology.run:FAIL