2023-08-31T17:58:18.135 INFO:root:teuthology version: 0.0.1.dev193+gc2f06e4 2023-08-31T17:58:18.136 DEBUG:teuthology.run:Teuthology command: teuthology --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} tasks/{0-install test/mon_thrash}} --owner scheduled_zack@teuthology --archive /home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946 --name zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi --verbose -- /home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946/orig.config.yaml 2023-08-31T17:58:18.176 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T17:58:18.257 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946 branch: main description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} tasks/{0-install test/mon_thrash}} email: null first_in_suite: false job_id: '7385946' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - mons down - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(TOO_FEW_PGS\) - \(OSD_SLOW_PING_TIME - slow request sha1: e1adcaefc485cd32098c363b5dee2d42ca2501d6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e1adcaefc485cd32098c363b5dee2d42ca2501d6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 9d7c18257836dc888b4a300f3b5af9f080910986 owner: scheduled_zack@teuthology priority: 1 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mds.a - mgr.x - osd.0 - osd.1 - - mon.b - mds.b - mon.c - mgr.y - osd.2 - osd.3 - - client.0 seed: 4542 sha1: e1adcaefc485cd32098c363b5dee2d42ca2501d6 sleep_before_teardown: 0 suite: smoke suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_9d7c18257836dc888b4a300f3b5af9f080910986/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 9d7c18257836dc888b4a300f3b5af9f080910986 targets: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD47Y3gGY+VCiy2zCUyPglTKagWSuA0MUjdp/qXWHkTu0mUlFOSxerS47brkG6TqJJ2h2Y0aR9/Bc9XBeFiCgEN+3FzCRpW6Ir5jozcUAS+ZegxsSYn/XdXr3W2VhHNw3qRL8HuW5HjJkb6fgxvCUZd1ydTNIsCtNifhHAjVBqW7+usNwsTlKrFVUGontmjR6rjwS/uG3rtArbKTMLR7/NKSK7wNnzzsK/lY1Y2hK2Shv/FrXgK3y5InJurobwZzzNW9a+pyFkCy7zjTg0c0siN9QkXU0wzw6AfMZj2qtIYJII7VXiYnKVoXJjFQCmiZe5RChOM9OA+PCu31aE1/Cgh3dbEj7Xll046CIoQpO6aQ7nWjwAatQeWZuqW1CU9VftuxcTULI7kjy0WeWxW8k42hKciBW6y/r29jwW5emnBweoGMUhniPZiZteiISvIpAi7bieubBIz3ZFXehI+UUXDICoiA91F3BKogo1MxC9Gvy5p10/OsGiP+D2tCcL4gM= smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD/L/qHCg9FeCU6xxrmcbb8o/ex6RLRMy4Cby/zPZ5An+HAaL4N4rzsRb9Jtbb8/sW+6nTFlDoEwtp6qwWZYZ1q3qgSJeeCbqyahNb1CQqzPt2vU7vzoK+FpU9R4otc1FZV/hug5Qitx49Mu2/PXv/AU+juGP6xKwBLhzl/GqILY5+cqZrrkfzKXMwqeGry5SRpMbjoweqQnERoa9tbWRCBXZapT3lTG04JgNGDm4KpXQPO4DPvVvt9YuhHqewaWfCjmfJLxEglcGJp4B/N+unkmBBeNmf/F2zD0bjXaqjRGBKKe+B7epr+drqGlYFXHxROWTQs2ODZpd47Qmsxnl94dJu07+68i+znOcswc66VvyBCC1yKMu1mMKnLx2uhrQduXbmAV4qkeHTnFbcNPehdVIC2qaHRgLsqTrlLaxb+pCrNAlNnZrKIlOmoI/1Hds6SvsJNDYE6oHFHldc4b9nmb9BFcA7x3ntdR1n/dd6fM+Q8KQfrR65xXWtmNhJH7E= smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUofrU5FHQ4mTdYzsjwSueQGxDzlHDlCl/VbOzdCx29hhz+U5h5rC5OKjOmPDjfnQjE3F2azQ/rJZuVWB29Q90PbZRdzK1D0DqVZTkOsndyTEYO943QxinMyLNa8KFrk44q2N/iUDF5/O7p9TySXz+FUg9Fpy7r0nAQupq1IHMh8O3pjvJm06Y6NAW6Vdgzn3pAwvqZM3Mdt4Vi92Wx/w8/MqQx7XLYAgJXNAJAcQIlHuxv2rK/LZI3lS48Q+9aIrPS/w1F/PfIiWvnAq6iB5YadjX29vggrK0ex57DaXC7nHI/jWxTBMIR3TzH+vha7cfo1Ohy4tvlTioPPBdAMcHfiqnQ9hWRie5nY4ucyN02NmYUh1FTDuHpzsr/h24kjpbey0M1RDSpr7M73OTMIFfnPi4/DGeFZxEUXagMSOkuI+0eJuBwaX2NrSz/OrhCmZrkV0zSJYmeSf71aoJ9o1OKJIq7Wv4ZFTgai8ePIy8lk9Vo4Ku3avOmedas1cNS4k= tasks: - install: cleanup: true - ceph: fs: xfs - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: paramiko-eoferror teuthology_sha1: c2f06e40df74acfa089b06a428d6af1ab9c07767 timestamp: 2023-08-31_17:43:55 tube: smithi user: zack verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-08-31T17:58:18.257 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_9d7c18257836dc888b4a300f3b5af9f080910986/qa; will attempt to use it 2023-08-31T17:58:18.258 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_9d7c18257836dc888b4a300f3b5af9f080910986/qa/tasks 2023-08-31T17:58:18.258 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-31T17:58:18.260 INFO:teuthology.task.internal:Checking packages... 2023-08-31T17:58:18.285 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e1adcaefc485cd32098c363b5dee2d42ca2501d6' 2023-08-31T17:58:18.285 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-31T17:58:18.286 INFO:teuthology.packaging:ref: None 2023-08-31T17:58:18.286 INFO:teuthology.packaging:tag: None 2023-08-31T17:58:18.287 INFO:teuthology.packaging:branch: main 2023-08-31T17:58:18.287 INFO:teuthology.packaging:sha1: e1adcaefc485cd32098c363b5dee2d42ca2501d6 2023-08-31T17:58:18.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-08-31T17:58:18.531 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5792.ge1adcaef 2023-08-31T17:58:18.532 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-31T17:58:18.538 INFO:teuthology.task.internal:no buildpackages task found 2023-08-31T17:58:18.539 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-31T17:58:18.552 INFO:teuthology.task.internal:Saving configuration 2023-08-31T17:58:18.567 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-31T17:58:18.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T17:58:18.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.189589', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkXqDPJ/kps1qfya8WbVnDFdD4S1ykTiZmF/y8ftJsJ6uIAbmJJPDgjFM9iYND9lN1mzC3qyc3rJs82BirTJkk='} 2023-08-31T17:58:18.620 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.191019', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFajBW7Cj/qRj6yruKIkp9smEWfYm3QyyFx20fxWlIXqVxnq8fKXV2IvVIz5eUsPcJeGou2cyfYIZopUsTWccyU='} 2023-08-31T17:58:18.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.192171', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJvk5H543bdtVXtB4D419fGXopzHd2U/SOoUM/Tj0wRKMbkcM83sXEm0nqiOtA2qhqpscNqd3HBa+6GJRR/PXeo='} 2023-08-31T17:58:18.643 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-31T17:58:18.649 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-08-31T17:58:18.649 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-08-31T17:58:18.650 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['client.0'] 2023-08-31T17:58:18.650 INFO:teuthology.run_tasks:Running task console_log... 2023-08-31T17:58:18.798 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbcf2173af0>, signals=[15]) 2023-08-31T17:58:18.799 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-31T17:58:18.805 INFO:teuthology.task.internal:Opening connections... 2023-08-31T17:58:18.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-08-31T17:58:18.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-31T17:58:18.827 ERROR:teuthology.orchestra.connection:Error authenticating with smithi002.front.sepia.ceph.com: No SSH private key found! 2023-08-31T17:58:18.828 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 421, in connect t.start_client(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 699, in start_client raise e File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError 2023-08-31T17:58:18.949 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a2a7bd1cd1344b20ad8a26e9e7413bd2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/task/internal/__init__.py", line 180, in connect rem.connect() File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/orchestra/remote.py", line 386, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 421, in connect t.start_client(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 699, in start_client raise e File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_c2f06e40df74acfa089b06a428d6af1ab9c07767/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError 2023-08-31T17:58:18.954 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-31T17:58:18.967 INFO:teuthology.nuke:Checking targets against current locks 2023-08-31T17:58:19.009 DEBUG:teuthology.nuke:shortname: smithi002 2023-08-31T17:58:19.010 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T17:58:19.038 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.189589', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkXqDPJ/kps1qfya8WbVnDFdD4S1ykTiZmF/y8ftJsJ6uIAbmJJPDgjFM9iYND9lN1mzC3qyc3rJs82BirTJkk='} 2023-08-31T17:58:19.045 DEBUG:teuthology.nuke:shortname: smithi031 2023-08-31T17:58:19.046 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T17:58:19.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.191019', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFajBW7Cj/qRj6yruKIkp9smEWfYm3QyyFx20fxWlIXqVxnq8fKXV2IvVIz5eUsPcJeGou2cyfYIZopUsTWccyU='} 2023-08-31T17:58:19.077 DEBUG:teuthology.nuke:shortname: smithi102 2023-08-31T17:58:19.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-31T17:58:19.107 INFO:teuthology.orchestra.console.smithi002:Power off 2023-08-31T17:58:19.108 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T17:58:19.123 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/zack-2023-08-31_17:43:55-smoke-main-distro-default-smithi/7385946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-31 17:53:25.192171', 'locked_by': 'scheduled_zack@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJvk5H543bdtVXtB4D419fGXopzHd2U/SOoUM/Tj0wRKMbkcM83sXEm0nqiOtA2qhqpscNqd3HBa+6GJRR/PXeo='} 2023-08-31T17:58:19.134 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-08-31T17:58:19.135 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:19.159 INFO:teuthology.orchestra.console.smithi031:Power off 2023-08-31T17:58:19.160 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T17:58:19.170 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-08-31T17:58:19.182 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-08-31T17:58:19.183 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:19.198 INFO:teuthology.orchestra.console.smithi102:Power off 2023-08-31T17:58:19.199 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-31T17:58:19.209 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-31T17:58:19.222 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2023-08-31T17:58:19.223 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:19.245 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2023-08-31T17:58:23.175 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:23.211 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:23.247 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:23.285 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-08-31T17:58:23.320 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-31T17:58:23.357 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2023-08-31T17:58:27.290 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:27.321 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:27.361 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:27.401 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-08-31T17:58:27.432 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-31T17:58:27.472 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2023-08-31T17:58:27.502 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-08-31T17:58:31.438 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:31.473 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-31T17:58:31.548 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-08-31T17:58:31.584 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2023-08-31T17:58:31.650 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-08-31T17:58:31.686 INFO:teuthology.orchestra.console.smithi102:Power off completed 2023-08-31T17:58:31.790 INFO:teuthology.run:Summary data: description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} tasks/{0-install test/mon_thrash}} failure_reason: '' owner: scheduled_zack@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a2a7bd1cd1344b20ad8a26e9e7413bd2 status: fail success: false 2023-08-31T17:58:31.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-31T17:58:31.868 INFO:teuthology.run:FAIL