2023-12-07T01:36:12.504 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-07T01:36:12.504 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979 --owner scheduled_yuriw@teuthology --description rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_host_drain} --verbose --name yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979/orig.config.yaml
2023-12-07T01:36:12.536 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-07T01:36:12.614 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979
  branch: wip-yuri-testing-2023-12-06-1240
  description: rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on
    mon_election/classic task/test_host_drain}
  email: yweinste@redhat.com
  first_in_suite: false
  job_id: '7480979'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  name: yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  os_type: rhel
  os_version: '8.6'
  overrides:
    admin_socket:
      branch: wip-yuri-testing-2023-12-06-1240
    ceph:
      conf:
        global:
          mon election default strategy: 1
        mgr:
          debug mgr: 20
          debug ms: 1
          mgr/cephadm/use_agent: true
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
        osd:
          debug ms: 1
          debug osd: 20
      flavor: default
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
    selinux:
      whitelist:
      - scontext=system_u:system_r:logrotate_t:s0
    workunit:
      branch: wip-yuri-testing-2023-12-06-1240
      sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
  owner: scheduled_yuriw@teuthology
  priority: 75
  repo: https://github.com/ceph/ceph-ci.git
  roles:
  - - host.a
    - mon.a
    - mgr.a
    - osd.0
    - osd.1
  - - host.b
    - mon.b
    - mgr.b
    - osd.2
    - osd.3
  - - host.c
    - mon.c
    - osd.4
    - osd.5
  seed: 6240
  sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
  sleep_before_teardown: 0
  subset: 111/120000
  suite: rados
  suite_branch: wip-yuri-testing-2023-12-06-1240
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f0c322331348463ad8afb13a3a03e833bee1c39c/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci.git
  suite_sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
  targets:
    smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
    smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
    smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=
  tasks:
  - pexec:
      all:
      - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup
      - sudo dnf -y  module reset container-tools
      - sudo dnf -y  module install container-tools:3.0 --allowerasing --nobest
      - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf
  - install: null
  - cephadm: null
  - cephadm.shell:
      host.a:
      - "set -ex\nHOSTNAMES=$(ceph orch host ls --format json | jq -r '.[] | .hostname')\n\
        for host in $HOSTNAMES; do\n  # find the hostname for \"host.c\" which will\
        \ have no mgr\n  HAS_MGRS=$(ceph orch ps --hostname ${host} --format json |\
        \ jq 'any(.daemon_type == \"mgr\")')\n  if [ \"$HAS_MGRS\" == \"false\" ]; then\n\
        \    HOST_C=\"${host}\"\n  fi\ndone\n# One last thing to worry about before\
        \ draining the host\n# is that the teuthology test tends to put the explicit\n\
        # hostnames in the placement for the mon service.\n# We want to make sure we\
        \ can drain without providing\n# --force and there is a check for the host being\
        \ removed\n# being listed explicitly in the placements. Therefore,\n# we should\
        \ remove it from the mon placement.\nceph orch ls mon --export > mon.yaml\n\
        sed /\"$HOST_C\"/d mon.yaml > mon_adjusted.yaml\nceph orch apply -i mon_adjusted.yaml\n\
        # now drain that host\nceph orch host drain $HOST_C --zap-osd-devices\n# wait\
        \ for drain to complete\nHOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\n\
        while [ \"$HOST_C_DAEMONS\" != \"No daemons reported\" ]; do\n  sleep 15\n \
        \ HOST_C_DAEMONS=$(ceph orch ps --hostname $HOST_C)\ndone\n# we want to check\
        \ the ability to remove the host from\n# the CRUSH map, so we should first verify\
        \ the host is in\n# the CRUSH map.\nceph osd getcrushmap -o compiled-crushmap\n\
        crushtool -d compiled-crushmap -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\n\
        if ! grep -q \"$HOST_C\" <<< \"$CRUSH_MAP\"; then\n  printf \"Expected to see\
        \ $HOST_C in CRUSH map. Saw:\\n\\n$CRUSH_MAP\"\n  exit 1\nfi\n# If the drain\
        \ was successful, we should be able to remove the\n# host without force with\
        \ no issues. If there are still daemons\n# we will get a response telling us\
        \ to drain the host and a\n# non-zero return code\nceph orch host rm $HOST_C\
        \ --rm-crush-entry\n# verify we've successfully removed the host from the CRUSH\
        \ map\nsleep 30\nceph osd getcrushmap -o compiled-crushmap\ncrushtool -d compiled-crushmap\
        \ -o crushmap.txt\nCRUSH_MAP=$(cat crushmap.txt)\nif grep -q \"$HOST_C\" <<<\
        \ \"$CRUSH_MAP\"; then\n  printf \"Saw $HOST_C in CRUSH map after it should\
        \ have been removed.\\n\\n$CRUSH_MAP\"\n  exit 1\nfi\n"
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269
  timestamp: 2023-12-07_00:12:58
  tube: smithi
  user: yuriw
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157
2023-12-07T01:36:12.614 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f0c322331348463ad8afb13a3a03e833bee1c39c/qa; will attempt to use it
2023-12-07T01:36:12.615 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f0c322331348463ad8afb13a3a03e833bee1c39c/qa/tasks
2023-12-07T01:36:12.615 INFO:teuthology.run_tasks:Running task internal.check_packages...
2023-12-07T01:36:12.617 INFO:teuthology.task.internal:Checking packages...
2023-12-07T01:36:12.640 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'f0c322331348463ad8afb13a3a03e833bee1c39c'
2023-12-07T01:36:12.640 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2023-12-07T01:36:12.640 INFO:teuthology.packaging:ref: None
2023-12-07T01:36:12.640 INFO:teuthology.packaging:tag: None
2023-12-07T01:36:12.640 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-12-06-1240
2023-12-07T01:36:12.640 INFO:teuthology.packaging:sha1: f0c322331348463ad8afb13a3a03e833bee1c39c
2023-12-07T01:36:12.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-12-06-1240
2023-12-07T01:36:12.776 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 101, in check_packages
    if package.version:
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 544, in version
    self._version = self._get_package_version()
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 955, in _get_package_version
    self.assert_result()
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 937, in assert_result
    raise VersionNotFoundError(self._result.url)
teuthology.exceptions.VersionNotFoundError: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-12-06-1240
2023-12-07T01:36:12.841 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5774b33ab33f40ed927fcae70b1fdaa0
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/internal/__init__.py", line 101, in check_packages
    if package.version:
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 544, in version
    self._version = self._get_package_version()
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 955, in _get_package_version
    self.assert_result()
  File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/packaging.py", line 937, in assert_result
    raise VersionNotFoundError(self._result.url)
teuthology.exceptions.VersionNotFoundError: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-12-06-1240
2023-12-07T01:36:12.849 INFO:teuthology.nuke:Checking targets against current locks
2023-12-07T01:36:12.881 DEBUG:teuthology.nuke:shortname: smithi081
2023-12-07T01:36:12.881 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-12-07T01:36:12.904 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 01:30:54.812463', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-12-07T01:36:12.909 DEBUG:teuthology.nuke:shortname: smithi088
2023-12-07T01:36:12.909 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-12-07T01:36:12.930 DEBUG:teuthology.nuke:shortname: smithi160
2023-12-07T01:36:12.930 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-12-07T01:36:12.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 01:30:54.815867', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-12-07T01:36:12.957 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_00:12:58-rados-wip-yuri-testing-2023-12-06-1240-distro-default-smithi/7480979', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 01:30:54.814288', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-12-07T01:36:12.969 INFO:teuthology.orchestra.console.smithi081:Power off
2023-12-07T01:36:12.970 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-07T01:36:12.987 INFO:teuthology.orchestra.console.smithi088:Power off
2023-12-07T01:36:12.987 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-07T01:36:13.000 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off
2023-12-07T01:36:13.000 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:13.012 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off
2023-12-07T01:36:13.012 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:13.026 INFO:teuthology.orchestra.console.smithi160:Power off
2023-12-07T01:36:13.030 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-07T01:36:13.039 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-12-07T01:36:13.039 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-12-07T01:36:13.052 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off
2023-12-07T01:36:13.052 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:13.075 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on
2023-12-07T01:36:17.042 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:17.052 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:17.077 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:17.154 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-12-07T01:36:17.163 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-12-07T01:36:17.187 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on
2023-12-07T01:36:21.158 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:21.166 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:21.189 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:21.268 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-12-07T01:36:21.277 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off
2023-12-07T01:36:21.301 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on
2023-12-07T01:36:21.369 INFO:teuthology.orchestra.console.smithi088:Power off completed
2023-12-07T01:36:21.378 INFO:teuthology.orchestra.console.smithi081:Power off completed
2023-12-07T01:36:25.304 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-07T01:36:25.413 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off
2023-12-07T01:36:25.514 INFO:teuthology.orchestra.console.smithi160:Power off completed
2023-12-07T01:36:25.618 INFO:teuthology.run:Summary data:
description: rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on
  mon_election/classic task/test_host_drain}
failure_reason: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-12-06-1240
owner: scheduled_yuriw@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5774b33ab33f40ed927fcae70b1fdaa0
status: fail
success: false

2023-12-07T01:36:25.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-07T01:36:25.708 INFO:teuthology.run:FAIL