2023-08-15T22:37:44.567 INFO:root:teuthology version: 0.0.1.dev183+g5435569
2023-08-15T22:37:44.568 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912 --name yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912/orig.config.yaml
2023-08-15T22:37:44.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-15T22:37:44.703 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912
  branch: wip-yuri7-testing-2023-08-15-0741-pacific
  description: fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd}
    distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health
    ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
  email: yweinste@redhat.com
  first_in_suite: false
  job_id: '7368912'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  log-rotate:
    ceph-mds: 10G
    ceph-osd: 10G
  machine_type: smithi
  name: yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  openstack:
  - volumes:
      count: 4
      size: 30
  - machine:
      disk: 200
  os_type: ubuntu
  os_version: '20.04'
  overrides:
    admin_socket:
      branch: wip-yuri7-testing-2023-08-15-0741-pacific
    ceph:
      conf:
        client:
          client mount timeout: 600
          debug client: 20
          debug ms: 1
          rados mon op timeout: 900
          rados osd op timeout: 900
        mds:
          debug mds: 20
          debug ms: 1
          mds debug frag: true
          mds debug scatterstat: true
          mds op complaint time: 180
          mds verify scatter: true
          osd op complaint time: 180
          rados mon op timeout: 900
          rados osd op timeout: 900
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon op complaint time: 120
        osd:
          bluestore block size: 96636764160
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          debug bluefs: 20
          debug bluestore: 20
          debug ms: 1
          debug osd: 20
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd failsafe full ratio: 0.95
          osd objectstore: bluestore
          osd op complaint time: 180
      flavor: default
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - overall HEALTH_
      - \(FS_DEGRADED\)
      - \(MDS_FAILED\)
      - \(MDS_DEGRADED\)
      - \(FS_WITH_FAILED_MDS\)
      - \(MDS_DAMAGE\)
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(FS_INLINE_DATA_DEPRECATED\)
      - overall HEALTH_
      - \(OSD_DOWN\)
      - \(OSD_
      - but it is still running
      - is not responding
      - evicting unresponsive client
      - POOL_APP_NOT_ENABLED
      - has not responded to cap revoke by MDS for over
      - MDS_CLIENT_LATE_RELEASE
      - responding to mclientcaps
      - RECENT_CRASH
      sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-yuri7-testing-2023-08-15-0741-pacific
      sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
  owner: scheduled_yuriw@teuthology
  priority: 75
  repo: https://github.com/ceph/ceph-ci.git
  roles:
  - - mon.a
    - mgr.y
    - mds.a
    - osd.0
    - osd.1
    - osd.2
    - osd.3
  - - mon.b
    - mon.c
    - mgr.x
    - mds.b
    - osd.4
    - osd.5
    - osd.6
    - osd.7
  - - client.0
  - - client.1
  - - client.2
  seed: 17
  sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
  sleep_before_teardown: 0
  subset: 111/120000
  suite: fs
  suite_branch: wip-yuri7-testing-2023-08-15-0741-pacific
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a73b87cee053ac22e6558c8cafa373095950c3c9/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci.git
  suite_sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
  targets:
    smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=
    smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0=
    smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0=
    smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=
    smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=
  tasks:
  - install:
      extra_packages:
        deb:
        - python3-cephfs
        - cephfs-shell
        - cephfs-top
        - cephfs-mirror
        rpm:
        - python3-cephfs
        - cephfs-top
        - cephfs-mirror
      extra_system_packages:
        deb:
        - bison
        - flex
        - libelf-dev
        - libssl-dev
        - network-manager
        - iproute2
        - util-linux
        - dump
        - indent
        - libaio-dev
        - libtool-bin
        - uuid-dev
        - xfslibs-dev
        rpm:
        - bison
        - flex
        - elfutils-libelf-devel
        - openssl-devel
        - NetworkManager
        - iproute
        - util-linux
        - libacl-devel
        - libaio-devel
        - libattr-devel
        - libtool
        - libuuid-devel
        - xfsdump
        - xfsprogs
        - xfsprogs-devel
        - libaio-devel
        - libtool
        - libuuid-devel
        - xfsprogs-devel
  - ceph: null
  - ceph-fuse: null
  - cephfs_test_runner:
      modules:
      - tasks.cephfs.test_misc
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 54355694caed5e99f46f0c9317b1e9bcf8940415
  timestamp: 2023-08-15_16:56:50
  tube: smithi
  user: yuriw
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2224
2023-08-15T22:37:44.704 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a73b87cee053ac22e6558c8cafa373095950c3c9/qa; will attempt to use it
2023-08-15T22:37:44.705 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a73b87cee053ac22e6558c8cafa373095950c3c9/qa/tasks
2023-08-15T22:37:44.705 INFO:teuthology.run_tasks:Running task internal.check_packages...
2023-08-15T22:37:44.707 INFO:teuthology.task.internal:Checking packages...
2023-08-15T22:37:44.735 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a73b87cee053ac22e6558c8cafa373095950c3c9'
2023-08-15T22:37:44.735 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2023-08-15T22:37:44.736 INFO:teuthology.packaging:ref: None
2023-08-15T22:37:44.736 INFO:teuthology.packaging:tag: None
2023-08-15T22:37:44.736 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-08-15-0741-pacific
2023-08-15T22:37:44.736 INFO:teuthology.packaging:sha1: a73b87cee053ac22e6558c8cafa373095950c3c9
2023-08-15T22:37:44.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri7-testing-2023-08-15-0741-pacific
2023-08-15T22:37:44.949 INFO:teuthology.task.internal:Found packages for ceph version 16.2.13-480-ga73b87ce-1focal
2023-08-15T22:37:44.949 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2023-08-15T22:37:44.955 INFO:teuthology.task.internal:no buildpackages task found
2023-08-15T22:37:44.956 INFO:teuthology.run_tasks:Running task internal.save_config...
2023-08-15T22:37:44.968 INFO:teuthology.task.internal:Saving configuration
2023-08-15T22:37:44.984 INFO:teuthology.run_tasks:Running task internal.check_lock...
2023-08-15T22:37:44.990 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:37:45.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.824874', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:37:45.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.828159', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:37:45.060 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368923', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:37:43.755973', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:37:45.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.823123', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:37:45.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.821012', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:37:45.102 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2023-08-15T22:37:45.109 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
2023-08-15T22:37:45.109 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-08-15T22:37:45.109 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['client.0']
2023-08-15T22:37:45.109 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['client.1']
2023-08-15T22:37:45.110 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['client.2']
2023-08-15T22:37:45.110 INFO:teuthology.run_tasks:Running task console_log...
2023-08-15T22:37:45.278 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7ff9f1236a90>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7ff9f0410e50>, signals=[15])
2023-08-15T22:37:45.280 INFO:teuthology.run_tasks:Running task internal.connect...
2023-08-15T22:37:45.294 INFO:teuthology.task.internal:Opening connections...
2023-08-15T22:37:45.295 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com
2023-08-15T22:37:45.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-15T22:37:45.347 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com
2023-08-15T22:37:45.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-15T22:37:45.404 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com
2023-08-15T22:37:45.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-15T22:38:45.406 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/task/internal/__init__.py", line 180, in connect
    rem.connect()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/remote.py", line 386, in connect
    self.ssh = connection.connect(**args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 349, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/util.py", line 283, in retry_on_signal
    return function()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 349, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 613, in connect
    self._wait(self._write_event)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: timed out
2023-08-15T22:38:45.475 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3e5385ea217845b0a2caed906d389260
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/task/internal/__init__.py", line 180, in connect
    rem.connect()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/remote.py", line 386, in connect
    self.ssh = connection.connect(**args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 349, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/util.py", line 283, in retry_on_signal
    return function()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 349, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 613, in connect
    self._wait(self._write_event)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: timed out
2023-08-15T22:38:45.478 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2023-08-15T22:38:45.492 INFO:teuthology.nuke:Checking targets against current locks
2023-08-15T22:38:49.046 DEBUG:teuthology.nuke:shortname: smithi100
2023-08-15T22:38:49.047 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:38:49.068 DEBUG:teuthology.nuke:shortname: smithi132
2023-08-15T22:38:49.069 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:38:49.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.824874', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:38:49.089 DEBUG:teuthology.nuke:shortname: smithi154
2023-08-15T22:38:49.090 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:38:49.095 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.828159', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:38:49.108 DEBUG:teuthology.nuke:shortname: smithi193
2023-08-15T22:38:49.108 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:38:49.123 INFO:teuthology.orchestra.console.smithi100:Power off
2023-08-15T22:38:49.124 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-15T22:38:49.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368923', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:37:43.755973', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:38:49.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.823123', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:38:49.145 DEBUG:teuthology.nuke:shortname: smithi196
2023-08-15T22:38:49.145 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-15T22:38:49.150 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off
2023-08-15T22:38:49.150 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:49.170 INFO:teuthology.orchestra.console.smithi132:Power off
2023-08-15T22:38:49.171 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-15T22:38:49.185 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368912', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:31:00.821012', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-08-15T22:38:49.187 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2023-08-15T22:38:49.193 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off
2023-08-15T22:38:49.194 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:49.216 INFO:teuthology.orchestra.console.smithi193:Power off
2023-08-15T22:38:49.217 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-15T22:38:49.239 INFO:teuthology.orchestra.console.smithi154:Power off
2023-08-15T22:38:49.240 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-15T22:38:49.253 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-08-15T22:38:49.254 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off
2023-08-15T22:38:49.254 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:49.295 INFO:teuthology.orchestra.console.smithi196:Power off
2023-08-15T22:38:49.296 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-15T22:38:49.308 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on
2023-08-15T22:38:49.320 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off
2023-08-15T22:38:49.320 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:49.347 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-08-15T22:38:53.189 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:53.254 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:53.299 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2023-08-15T22:38:53.309 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:53.349 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:53.366 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-08-15T22:38:53.420 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on
2023-08-15T22:38:53.460 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-08-15T22:38:54.057 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off
2023-08-15T22:38:54.057 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:54.081 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off
2023-08-15T22:38:54.182 INFO:teuthology.orchestra.console.smithi154:Power off completed
2023-08-15T22:38:57.303 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:57.367 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:57.421 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:57.461 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:38:58.414 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-08-15T22:38:58.479 DEBUG:teuthology.orchestra.console.smithi132:check power output: 
2023-08-15T22:38:58.531 DEBUG:teuthology.orchestra.console.smithi193:check power output: 
2023-08-15T22:38:58.573 DEBUG:teuthology.orchestra.console.smithi196:check power output: 
2023-08-15T22:39:02.415 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:02.480 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:02.527 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2023-08-15T22:39:02.532 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:02.574 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:02.628 INFO:teuthology.orchestra.console.smithi100:Power off completed
2023-08-15T22:39:03.593 DEBUG:teuthology.orchestra.console.smithi132:check power output: 
2023-08-15T22:39:03.643 DEBUG:teuthology.orchestra.console.smithi193:check power output: 
2023-08-15T22:39:03.685 DEBUG:teuthology.orchestra.console.smithi196:check power output: 
2023-08-15T22:39:07.594 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:07.644 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:07.686 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:08.708 DEBUG:teuthology.orchestra.console.smithi132:check power output: 
2023-08-15T22:39:08.757 DEBUG:teuthology.orchestra.console.smithi193:check power output: 
2023-08-15T22:39:08.798 DEBUG:teuthology.orchestra.console.smithi196:check power output: 
2023-08-15T22:39:12.710 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:12.758 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:12.800 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:12.869 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off
2023-08-15T22:39:12.970 INFO:teuthology.orchestra.console.smithi193:Power off completed
2023-08-15T22:39:13.821 DEBUG:teuthology.orchestra.console.smithi132:check power output: 
2023-08-15T22:39:13.911 DEBUG:teuthology.orchestra.console.smithi196:check power output: 
2023-08-15T22:39:17.823 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:17.913 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:17.933 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off
2023-08-15T22:39:18.035 INFO:teuthology.orchestra.console.smithi132:Power off completed
2023-08-15T22:39:19.024 DEBUG:teuthology.orchestra.console.smithi196:check power output: 
2023-08-15T22:39:23.026 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-15T22:39:23.137 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off
2023-08-15T22:39:23.238 INFO:teuthology.orchestra.console.smithi196:Power off completed
2023-08-15T22:39:23.342 INFO:teuthology.run:Summary data:
description: fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd}
  distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health
  ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
failure_reason: timed out
owner: scheduled_yuriw@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3e5385ea217845b0a2caed906d389260
status: fail
success: false

2023-08-15T22:39:23.343 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-15T22:39:28.424 INFO:teuthology.run:FAIL