2024-06-10T06:08:12.604 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc
2024-06-10T06:08:12.604 DEBUG:teuthology.run:Teuthology command: teuthology --description upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} --verbose --archive /home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827 --owner scheduled_teuthology@teuthology --name teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827/orig.config.yaml
2024-06-10T06:08:12.650 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-06-10T06:08:12.763 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827
  branch: quincy
  description: upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8}
  email: ceph-qa@ceph.com
  first_in_suite: false
  job_id: '7747827'
  kernel:
    kdb: 1
    sha1: distro
  ktype: distro
  last_in_suite: false
  machine_type: smithi
  meta:
  - desc: 'Run ceph on two nodes, using one of them as a client,
  
      with a separate client-only node.
  
      Use xfs beneath the osds.
  
      install ceph/quincy v17.2.5 and the v17.2.x point versions
  
      run workload and upgrade-sequence in parallel
  
      (every point release should be tested)
  
      run workload and upgrade-sequence in parallel
  
      install ceph/quincy  latest version
  
      run workload and upgrade-sequence in parallel
  
      Overall upgrade path is - quincy-latest.point-1 => quincy-latest.point => quincy-latest
  
      '
  name: teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi
  no_nested_subset: false
  openstack:
  - volumes:
      count: 3
      size: 30
  os_type: centos
  os_version: 8.stream
  overrides:
    admin_socket:
      branch: quincy
    ceph:
      conf:
        client:
          rgw crypt s3 kms backend: testing
          rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=
            testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=
          rgw_crypt_require_ssl: false
        global:
          mon_mds_skip_sanity: true
          mon_warn_on_pool_no_app: false
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon debug unsafe allow tier with nonempty snaps: true
        osd:
          debug ms: 1
          debug osd: 20
          osd map max advance: 1000
          osd_class_default_list: '*'
          osd_class_load_list: '*'
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - reached quota
      - scrub
      - osd_map_max_advance
      - wrongly marked
      - FS_DEGRADED
      - POOL_APP_NOT_ENABLED
      - CACHE_POOL_NO_HIT_SET
      - POOL_FULL
      - SMALLER_PG
      - pool\(s\) full
      - OSD_DOWN
      - missing hit_sets
      - CACHE_POOL_NEAR_FULL
      - PG_AVAILABILITY
      - PG_DEGRADED
      - application not enabled
      - cache pools at or near target size
      - filesystem is degraded
      - OBJECT_MISPLACED
      sha1: 658e3c7068357222a961b3107ed1c91a5ab3a893
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: 658e3c7068357222a961b3107ed1c91a5ab3a893
    selinux:
      whitelist:
      - scontext=system_u:system_r:logrotate_t:s0
    workunit:
      branch: quincy
      sha1: 18c8f8967444b952c83b736d71c12da15d068318
  owner: scheduled_teuthology@teuthology
  priority: 820
  repo: https://git.ceph.com/ceph.git
  roles:
  - - mon.a
    - mds.a
    - osd.0
    - osd.1
    - osd.2
    - mgr.x
  - - mon.b
    - mon.c
    - osd.3
    - osd.4
    - osd.5
    - client.0
  - - client.1
  seed: 4928
  sha1: 658e3c7068357222a961b3107ed1c91a5ab3a893
  sleep_before_teardown: 0
  subset: 0/1
  suite: upgrade:quincy-p2p
  suite_branch: quincy
  suite_path: /home/teuthworker/src/git.ceph.com_ceph_18c8f8967444b952c83b736d71c12da15d068318/qa
  suite_relpath: qa
  suite_repo: https://git.ceph.com/ceph.git
  suite_sha1: 18c8f8967444b952c83b736d71c12da15d068318
  targets:
    smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5059oaS5+IKJ/VwgBIn9/+Qp0rvjACXJqvzh1ZzXjpABl8gzRVGAsRq20rDxTh3y2CYluiVfuyqaKPPW2Tuyo=
    smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbnV3AYbZVR4AvdBjaU6J7gzeZoLT/PylstO88wuJsqSalexIidznkdyUnbtuo82qVQ7Yln9B9b5FOGsNMYteQ=
    smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMEm2OnFeSoOqNFbFgv1pt8yPRkLBa+3icecGHVU9f4p9/Ij7pbndD9Ltph67/s5BNwokpzByaji0oLUYrLlzU=
  tasks:
  - print: '****  done quincy about to install v17.2.0 '
  - install:
      tag: v17.2.0
  - print: '**** done v17.2.0 install'
  - ceph:
      add_osds_to_crush: true
      fs: xfs
  - print: '**** done ceph xfs'
  - sequential:
    - workload
  - print: '**** done workload v17.2.0'
  - install.upgrade:
      mon.a:
        tag: v17.2.5
      mon.b:
        tag: v17.2.5
  - parallel:
    - workload_quincy
    - upgrade-sequence_quincy
  - print: '**** done parallel quincy v17.2.5'
  - install.upgrade:
      mon.a: null
      mon.b: null
  - parallel:
    - workload_quincy
    - upgrade-sequence_quincy
  - print: '**** done parallel quincy branch'
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa
  timestamp: 2024-06-10_00:40:02
  tube: smithi
  upgrade-sequence_quincy:
    sequential:
    - print: '**** done branch: quincy install.upgrade'
    - ceph.restart:
      - mds.a
    - sleep:
        duration: 60
    - ceph.restart:
      - osd.0
    - sleep:
        duration: 30
    - ceph.restart:
      - osd.1
    - sleep:
        duration: 30
    - ceph.restart:
      - osd.2
    - sleep:
        duration: 30
    - ceph.restart:
      - osd.3
    - sleep:
        duration: 30
    - ceph.restart:
      - osd.4
    - sleep:
        duration: 30
    - ceph.restart:
      - osd.5
    - sleep:
        duration: 60
    - ceph.restart:
      - mgr.x
    - sleep:
        duration: 60
    - ceph.restart:
      - mon.a
    - sleep:
        duration: 60
    - ceph.restart:
      - mon.b
    - sleep:
        duration: 60
    - ceph.restart:
      - mon.c
    - sleep:
        duration: 60
    - print: '**** done ceph.restart all quincy branch mds/osd/mon'
  user: teuthology
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841
  workload:
    sequential:
    - workunit:
        clients:
          client.0:
          - suites/blogbench.sh
  workload_quincy:
    full_sequential:
    - workunit:
        branch: quincy
        clients:
          client.1:
          - rados/test.sh
          - cls
        env:
          CLS_RBD_GTEST_FILTER: '*:-TestClsRbd.mirror_snapshot'
    - print: '**** done rados/test.sh &  cls workload_quincy'
    - sequential:
      - rgw:
        - client.0
      - print: '**** done rgw workload_quincy'
      - rbd_fsx:
          clients:
          - client.0
          size: 134217728
      - print: '**** done rbd_fsx workload_quincy'
2024-06-10T06:08:12.763 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_18c8f8967444b952c83b736d71c12da15d068318/qa; will attempt to use it
2024-06-10T06:08:12.764 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_18c8f8967444b952c83b736d71c12da15d068318/qa/tasks
2024-06-10T06:08:12.764 INFO:teuthology.run_tasks:Running task internal.check_packages...
2024-06-10T06:08:12.766 INFO:teuthology.task.internal:Checking packages...
2024-06-10T06:08:12.787 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '658e3c7068357222a961b3107ed1c91a5ab3a893'
2024-06-10T06:08:12.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2024-06-10T06:08:12.787 INFO:teuthology.packaging:ref: None
2024-06-10T06:08:12.787 INFO:teuthology.packaging:tag: None
2024-06-10T06:08:12.787 INFO:teuthology.packaging:branch: quincy
2024-06-10T06:08:12.787 INFO:teuthology.packaging:sha1: 658e3c7068357222a961b3107ed1c91a5ab3a893
2024-06-10T06:08:12.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy
2024-06-10T06:08:12.984 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-980.g658e3c70
2024-06-10T06:08:12.986 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2024-06-10T06:08:12.993 INFO:teuthology.task.internal:no buildpackages task found
2024-06-10T06:08:12.993 INFO:teuthology.run_tasks:Running task internal.save_config...
2024-06-10T06:08:13.013 INFO:teuthology.task.internal:Saving configuration
2024-06-10T06:08:13.030 INFO:teuthology.run_tasks:Running task internal.check_lock...
2024-06-10T06:08:13.036 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-06-10T06:08:13.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-10 06:02:46.621241', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5059oaS5+IKJ/VwgBIn9/+Qp0rvjACXJqvzh1ZzXjpABl8gzRVGAsRq20rDxTh3y2CYluiVfuyqaKPPW2Tuyo='}
2024-06-10T06:08:13.082 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-10 06:02:46.622846', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbnV3AYbZVR4AvdBjaU6J7gzeZoLT/PylstO88wuJsqSalexIidznkdyUnbtuo82qVQ7Yln9B9b5FOGsNMYteQ='}
2024-06-10T06:08:13.102 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-10_00:40:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7747827', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-10 06:02:46.624229', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMEm2OnFeSoOqNFbFgv1pt8yPRkLBa+3icecGHVU9f4p9/Ij7pbndD9Ltph67/s5BNwokpzByaji0oLUYrLlzU='}
2024-06-10T06:08:13.102 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2024-06-10T06:08:13.109 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'mgr.x']
2024-06-10T06:08:13.110 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.b', 'mon.c', 'osd.3', 'osd.4', 'osd.5', 'client.0']
2024-06-10T06:08:13.110 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['client.1']
2024-06-10T06:08:13.110 INFO:teuthology.run_tasks:Running task console_log...
2024-06-10T06:08:13.190 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f25a53f31f0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f25a56a3be0>, signals=[15])
2024-06-10T06:08:13.190 INFO:teuthology.run_tasks:Running task internal.connect...
2024-06-10T06:08:13.196 INFO:teuthology.task.internal:Opening connections...
2024-06-10T06:08:13.196 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com
2024-06-10T06:08:13.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-06-10T06:08:13.272 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com
2024-06-10T06:08:13.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-06-10T06:08:13.349 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com
2024-06-10T06:08:13.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-06-10T06:08:13.424 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2024-06-10T06:08:13.431 DEBUG:teuthology.orchestra.run.smithi029:> uname -m
2024-06-10T06:08:13.454 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2024-06-10T06:08:13.454 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 8"
2024-06-10T06:08:13.516 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31"
2024-06-10T06:08:13.517 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-06-10T06:08:13.517 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/"
2024-06-10T06:08:13.517 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-06-10T06:08:13.517 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-06-10T06:08:13.517 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-06-10T06:08:13.517 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2024-06-10T06:08:13.534 DEBUG:teuthology.orchestra.run.smithi046:> uname -m
2024-06-10T06:08:13.557 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64
2024-06-10T06:08:13.557 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 8"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31"
2024-06-10T06:08:13.619 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-06-10T06:08:13.620 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/"
2024-06-10T06:08:13.620 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-06-10T06:08:13.620 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-06-10T06:08:13.620 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-06-10T06:08:13.620 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server
2024-06-10T06:08:13.634 DEBUG:teuthology.orchestra.run.smithi148:> uname -m
2024-06-10T06:08:13.657 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64
2024-06-10T06:08:13.657 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release
2024-06-10T06:08:13.718 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream"
2024-06-10T06:08:13.718 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8"
2024-06-10T06:08:13.718 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos"
2024-06-10T06:08:13.718 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora"
2024-06-10T06:08:13.718 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-06-10T06:08:13.719 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-06-10T06:08:13.721 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server
2024-06-10T06:08:13.743 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2024-06-10T06:08:13.751 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2024-06-10T06:08:13.755 INFO:teuthology.task.internal:Checking for old test directory...
2024-06-10T06:08:13.756 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest
2024-06-10T06:08:13.758 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest
2024-06-10T06:08:13.761 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest
2024-06-10T06:08:13.782 INFO:teuthology.run_tasks:Running task internal.check_ceph_data...
2024-06-10T06:08:13.788 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph...
2024-06-10T06:08:13.788 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph)
2024-06-10T06:08:13.821 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph)
2024-06-10T06:08:13.823 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph)
2024-06-10T06:08:13.863 INFO:teuthology.run_tasks:Running task internal.vm_setup...
2024-06-10T06:08:13.932 INFO:teuthology.run_tasks:Running task kernel...
2024-06-10T06:08:13.945 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'}
2024-06-10T06:08:13.945 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.1': {'kdb': 1, 'sha1': 'distro'}}, timeout 300
2024-06-10T06:08:13.946 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv
2024-06-10T06:08:13.946 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv
2024-06-10T06:08:13.947 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv
2024-06-10T06:08:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-06-10T06:08:13.968 DEBUG:teuthology.orchestra.run.smithi046:> uname -r
2024-06-10T06:08:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-06-10T06:08:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-06-10T06:08:13.970 DEBUG:teuthology.orchestra.run.smithi029:> uname -r
2024-06-10T06:08:13.970 DEBUG:teuthology.orchestra.run.smithi148:> uname -r
2024-06-10T06:08:14.030 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-489.el8.x86_64
2024-06-10T06:08:14.030 INFO:teuthology.task.kernel:Running kernel on smithi046: 4.18.0-489.el8.x86_64
2024-06-10T06:08:14.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel
2024-06-10T06:08:14.031 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-489.el8.x86_64
2024-06-10T06:08:14.032 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-489.el8.x86_64
2024-06-10T06:08:14.032 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel
2024-06-10T06:08:14.033 INFO:teuthology.orchestra.run.smithi029.stdout:4.18.0-489.el8.x86_64
2024-06-10T06:08:14.033 INFO:teuthology.task.kernel:Running kernel on smithi029: 4.18.0-489.el8.x86_64
2024-06-10T06:08:14.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel
2024-06-10T06:08:16.649 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 8 - AppStream                     416  B/s |  38  B     00:00
2024-06-10T06:08:16.685 INFO:teuthology.orchestra.run.smithi148.stderr:Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
2024-06-10T06:08:16.706 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 8 - AppStream                     544  B/s |  38  B     00:00
2024-06-10T06:08:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-06-10T06:08:16.721 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 1236, in task
    with parallel() as p:
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 1270, in process_role
    version = need_to_install_distro(role_remote, role_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 761, in need_to_install_distro
    install_stdout = remote.sh(
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 97, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'sudo yum install -y kernel'
2024-06-10T06:08:16.888 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a152d1e5d3fb41fc9294a18524068c61
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 1236, in task
    with parallel() as p:
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 1270, in process_role
    version = need_to_install_distro(role_remote, role_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/kernel.py", line 761, in need_to_install_distro
    install_stdout = remote.sh(
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 97, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/remote.py", line 523, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 1: 'sudo yum install -y kernel'
2024-06-10T06:08:16.891 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-06-10T06:08:16.899 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2024-06-10T06:08:17.048 INFO:teuthology.run:Summary data:
description: upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8}
failure_reason: 'Command failed on smithi148 with status 1: ''sudo yum install -y
  kernel'''
owner: scheduled_teuthology@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a152d1e5d3fb41fc9294a18524068c61
status: fail
success: false

2024-06-10T06:08:17.048 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-06-10T06:08:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-06-10T06:08:17.063 INFO:teuthology.orchestra.run.smithi046.stderr:Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
2024-06-10T06:08:17.131 INFO:teuthology.run:FAIL