2022-11-27T17:34:23.479 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-11-27T17:34:23.482 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} --name yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721 -- /home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721/orig.config.yaml 2022-11-27T17:34:23.546 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-11-27T17:34:23.631 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721 branch: main description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} email: ceph-qa@ceph.io first_in_suite: false job_id: '7088721' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - RECENT_CRASH sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: dry_run: true install: ceph: flavor: default sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 sleep_before_teardown: 0 suite: fs suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAXQcBt9Pf4a4QsDxb/ldpDGiknTr5vOmmVVCzZWbXpmgIEtPEo5BUZ4ovE/y067yVpV1fTWLztHqtipJtdWedo= smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEk6lqfLAicZY/cjf7eoGecbzwL+Frnvws+9uajk15RFsc1WaF9XwyBdeFdT0JY9zL1bEmFAMYoH1mBwL35icxy05ODKWD9m6oTu6Fb/bGeQHOtklmI373ymhJ8Vce8hoLRaQyIeJKKvusGxlkhaAszcbFtR+oBd11TqyuGe7IzO7254H0xKNjqbn7sc+oOK/3q6Bf8Xwzp6gSLzrlLVWxJBb1g/tS/GwuxX8Eu5kIX+oLSpPdwQrtZY2YYrqsjtV9ta3MonN5JlnvwQsa6NIrcynmlf5a0ri6XX+wYT2jZFKmJepz+NMXkyX9PksEfynIW4Vk2qBzP7zIeqJWXuTdew7f5lknTJvkXNqCplqVYqiAw3hijn2ajl5ycjJKvcbMWeFSCSE2sSmHHuf4ywCDQiukBO+GaWq2OI1butKcSPjotXADA46DHV9lTqV1pTopj9ALnyf/byQc3PucAUKJWspGhd3d6uD/xI2g47o4qWcLiNeWVgP9KfqC5IlGcFU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_snapshots teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 timestamp: 2022-11-23_17:24:26 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20399 2022-11-27T17:34:23.632 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa; will attempt to use it 2022-11-27T17:34:23.633 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa/tasks 2022-11-27T17:34:23.634 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-11-27T17:34:23.642 INFO:teuthology.task.internal:Checking packages... 2022-11-27T17:34:23.681 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '697b0d99d9a52df92b20c8e053dcd46c75f364a9' 2022-11-27T17:34:23.682 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-11-27T17:34:23.682 INFO:teuthology.packaging:ref: None 2022-11-27T17:34:23.683 INFO:teuthology.packaging:tag: None 2022-11-27T17:34:23.683 INFO:teuthology.packaging:branch: main 2022-11-27T17:34:23.683 INFO:teuthology.packaging:sha1: 697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-27T17:34:23.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=main 2022-11-27T17:34:23.929 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-995.gaf6ed506 2022-11-27T17:34:23.930 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-11-27T17:34:23.937 INFO:teuthology.task.internal:no buildpackages task found 2022-11-27T17:34:23.938 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-11-27T17:34:23.958 INFO:teuthology.task.internal:Saving configuration 2022-11-27T17:34:23.983 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-11-27T17:34:23.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-11-27T17:34:24.020 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-11-27 17:29:19.102342', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAXQcBt9Pf4a4QsDxb/ldpDGiknTr5vOmmVVCzZWbXpmgIEtPEo5BUZ4ovE/y067yVpV1fTWLztHqtipJtdWedo='} 2022-11-27T17:34:24.052 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-11-27 17:29:19.101006', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMO4sYvvmCk+mxu49oOfkuhURmU9R+iE2Qk74qA5Io+na5rW+8XXTNFVi+zkdvicE/hD2gD0kxoXelu5V9BLJwU='} 2022-11-27T17:34:24.053 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-11-27T17:34:24.060 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2022-11-27T17:34:24.060 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2022-11-27T17:34:24.061 INFO:teuthology.run_tasks:Running task console_log... 2022-11-27T17:34:24.152 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc7e8468840>, signals=[15]) 2022-11-27T17:34:24.154 INFO:teuthology.run_tasks:Running task internal.connect... 2022-11-27T17:34:24.163 INFO:teuthology.task.internal:Opening connections... 2022-11-27T17:34:24.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:34:24.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-27T17:34:24.249 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:34:24.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-27T17:34:24.360 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-11-27T17:34:24.367 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2022-11-27T17:34:24.389 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2022-11-27T17:34:24.390 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2022-11-27T17:34:24.449 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Red Hat Enterprise Linux" 2022-11-27T17:34:24.449 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8.6 (Ootpa)" 2022-11-27T17:34:24.450 INFO:teuthology.orchestra.run.smithi099.stdout:ID="rhel" 2022-11-27T17:34:24.450 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="fedora" 2022-11-27T17:34:24.451 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8.6" 2022-11-27T17:34:24.451 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2022-11-27T17:34:24.452 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-11-27T17:34:24.452 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2022-11-27T17:34:24.452 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-11-27T17:34:24.453 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.redhat.com/" 2022-11-27T17:34:24.453 INFO:teuthology.orchestra.run.smithi099.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-11-27T17:34:24.454 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-11-27T17:34:24.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:24.455 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-11-27T17:34:24.456 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-11-27T17:34:24.456 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-11-27T17:34:24.456 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-11-27T17:34:24.458 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-11-27T17:34:24.480 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2022-11-27T17:34:24.501 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2022-11-27T17:34:24.501 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2022-11-27T17:34:24.561 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Red Hat Enterprise Linux" 2022-11-27T17:34:24.562 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8.6 (Ootpa)" 2022-11-27T17:34:24.562 INFO:teuthology.orchestra.run.smithi119.stdout:ID="rhel" 2022-11-27T17:34:24.562 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="fedora" 2022-11-27T17:34:24.563 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8.6" 2022-11-27T17:34:24.563 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2022-11-27T17:34:24.564 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-11-27T17:34:24.564 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2022-11-27T17:34:24.564 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-11-27T17:34:24.565 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.redhat.com/" 2022-11-27T17:34:24.565 INFO:teuthology.orchestra.run.smithi119.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-11-27T17:34:24.565 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-11-27T17:34:24.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:24.566 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-11-27T17:34:24.567 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-11-27T17:34:24.567 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-11-27T17:34:24.568 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-11-27T17:34:24.568 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-11-27T17:34:24.591 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-11-27T17:34:24.604 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-11-27T17:34:24.612 INFO:teuthology.task.internal:Checking for old test directory... 2022-11-27T17:34:24.613 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2022-11-27T17:34:24.616 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2022-11-27T17:34:24.636 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-11-27T17:34:24.650 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-11-27T17:34:24.650 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2022-11-27T17:34:24.673 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2022-11-27T17:34:24.707 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-11-27T17:34:24.765 INFO:teuthology.run_tasks:Running task kernel... 2022-11-27T17:34:24.782 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2022-11-27T17:34:24.783 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2022-11-27T17:34:24.784 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2022-11-27T17:34:24.784 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2022-11-27T17:34:24.785 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2022-11-27T17:34:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-27T17:34:24.804 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2022-11-27T17:34:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-27T17:34:24.806 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2022-11-27T17:34:24.863 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:24.864 INFO:teuthology.task.kernel:Running kernel on smithi119: 4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:24.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2022-11-27T17:34:24.865 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:24.866 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:24.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2022-11-27T17:34:25.785 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:25.785 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2022-11-27T17:34:25.888 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:25.889 INFO:teuthology.orchestra.run.smithi119.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:25.889 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:25.952 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:25.953 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2022-11-27T17:34:26.022 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:26.022 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:26.023 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:29.124 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-11-27T17:34:29.804 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2022-11-27T17:34:31.949 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 13 MB 00:02 2022-11-27T17:34:32.629 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 13 MB 00:02 2022-11-27T17:34:34.924 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 482 kB/s | 24 kB 00:00 2022-11-27T17:34:35.592 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2022-11-27T17:34:35.637 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 307 kB/s | 24 kB 00:00 2022-11-27T17:34:35.639 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-11-27T17:34:35.640 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2022-11-27T17:34:35.640 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-11-27T17:34:35.775 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2022-11-27T17:34:36.104 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:36.105 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2022-11-27T17:34:36.149 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:36.150 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:36.150 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:36.319 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2022-11-27T17:34:36.365 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2022-11-27T17:34:36.369 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2022-11-27T17:34:36.370 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2022-11-27T17:34:36.387 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:02 ago on Sun 27 Nov 2022 05:34:34 PM UTC. 2022-11-27T17:34:36.431 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-11-27T17:34:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2022-11-27T17:34:36.482 DEBUG:teuthology.orchestra.run.smithi119:> echo no | sudo yum reinstall kernel || true 2022-11-27T17:34:36.511 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2022-11-27T17:34:36.813 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:36.813 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2022-11-27T17:34:36.840 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:36.840 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2022-11-27T17:34:36.853 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:36.853 INFO:teuthology.orchestra.run.smithi119.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:36.854 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:36.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:36.886 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:36.886 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:34:37.090 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:02 ago on Sun 27 Nov 2022 05:34:35 PM UTC. 2022-11-27T17:34:37.118 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:03 ago on Sun 27 Nov 2022 05:34:34 PM UTC. 2022-11-27T17:34:37.131 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-11-27T17:34:37.155 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-11-27T17:34:37.164 INFO:teuthology.orchestra.run.smithi119.stderr:Error: No packages marked for reinstall. 2022-11-27T17:34:37.190 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2022-11-27T17:34:37.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum reinstall -y kernel || true 2022-11-27T17:34:37.241 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel --last | head -n 1 2022-11-27T17:34:37.320 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2022-11-27T17:34:37.322 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:37.322 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-11-27T17:34:37.323 DEBUG:teuthology.parallel:result is None 2022-11-27T17:34:37.540 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2022-11-27T17:34:37.541 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2022-11-27T17:34:37.583 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:37.583 INFO:teuthology.orchestra.run.smithi119.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-11-27T17:34:37.584 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:34:37.818 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:02 ago on Sun 27 Nov 2022 05:34:35 PM UTC. 2022-11-27T17:34:37.858 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2022-11-27T17:34:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:Error: No packages marked for reinstall. 2022-11-27T17:34:37.937 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel --last | head -n 1 2022-11-27T17:34:38.017 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2022-11-27T17:34:38.018 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2022-11-27T17:34:38.019 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-11-27T17:34:38.019 DEBUG:teuthology.parallel:result is None 2022-11-27T17:34:38.019 INFO:teuthology.run_tasks:Running task internal.base... 2022-11-27T17:34:38.027 INFO:teuthology.task.internal:Creating test directory... 2022-11-27T17:34:38.028 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-11-27T17:34:38.032 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-11-27T17:34:38.049 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-11-27T17:34:38.057 INFO:teuthology.run_tasks:Running task internal.archive... 2022-11-27T17:34:38.063 INFO:teuthology.task.internal:Creating archive directory... 2022-11-27T17:34:38.064 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-11-27T17:34:38.091 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-11-27T17:34:38.140 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-11-27T17:34:38.149 INFO:teuthology.task.internal:Enabling coredump saving... 2022-11-27T17:34:38.150 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-11-27T17:34:38.173 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-11-27T17:34:38.209 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-27T17:34:38.210 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-27T17:34:38.221 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-27T17:34:38.223 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-11-27T17:34:38.224 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-11-27T17:34:38.231 INFO:teuthology.task.internal:Configuring sudo... 2022-11-27T17:34:38.232 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-11-27T17:34:38.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-11-27T17:34:38.293 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-11-27T17:34:38.312 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-11-27T17:34:38.313 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-11-27T17:34:38.335 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-11-27T17:34:38.352 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-27T17:34:38.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-27T17:34:38.504 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-27T17:34:38.551 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-27T17:34:38.592 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:34:38.593 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-11-27T17:34:38.661 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-27T17:34:38.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-11-27T17:34:38.757 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-27T17:34:38.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-11-27T17:34:38.882 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:34:38.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-11-27T17:34:38.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2022-11-27T17:34:38.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2022-11-27T17:34:39.007 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-27T17:34:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-11-27T17:34:39.433 INFO:teuthology.run_tasks:Running task internal.timer... 2022-11-27T17:34:39.445 INFO:teuthology.task.internal:Starting timer... 2022-11-27T17:34:39.446 INFO:teuthology.run_tasks:Running task pcp... 2022-11-27T17:34:39.466 INFO:teuthology.run_tasks:Running task selinux... 2022-11-27T17:34:39.476 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-11-27T17:34:39.477 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2022-11-27T17:34:39.535 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2022-11-27T17:34:39.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2022-11-27T17:34:39.586 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2022-11-27T17:34:39.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-11-27T17:34:39.588 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2022-11-27T17:34:39.614 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2022-11-27T17:34:39.615 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2022-11-27T17:34:39.636 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2022-11-27T17:34:39.637 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2022-11-27T17:34:39.638 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-27T17:34:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-27T17:34:39.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-11-27T17:34:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-11-27T17:34:39.711 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-11-27T17:34:39.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2022-11-27T17:34:39.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2022-11-27T17:34:39.793 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-11-27T17:34:39.806 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2022-11-27T17:34:39.814 INFO:teuthology.repo_utils:Fetching main from origin 2022-11-27T17:34:39.983 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2022-11-27T17:34:40.012 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-11-27T17:34:40.015 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2022-11-27T17:42:42.253 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2022-11-27T17:42:42.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-11-27T17:42:42.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-27T17:42:42.340 DEBUG:teuthology.orchestra.run.smithi099:> true 2022-11-27T17:42:42.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-11-27T17:42:42.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-11-27T17:42:42.485 DEBUG:teuthology.orchestra.run.smithi119:> true 2022-11-27T17:42:42.553 INFO:teuthology.run_tasks:Running task clock... 2022-11-27T17:42:42.568 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-11-27T17:42:42.569 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-11-27T17:42:42.571 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-11-27T17:42:42.574 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-11-27T17:42:42.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-11-27T17:42:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-11-27T17:42:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-11-27T17:42:42.627 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-11-27T17:42:42.641 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-11-27T17:42:42.681 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2022-11-27T17:42:42.692 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2022-11-27T17:42:42.706 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-11-27T17:42:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-11-27T17:42:42.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2022-11-27T17:42:42.732 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2022-11-27T17:42:42.745 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-11-27T17:42:42.759 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-11-27T17:42:42.786 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2022-11-27T17:42:42.789 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-11-27T17:42:42.790 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2022-11-27T17:42:42.790 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.790 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.790 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.791 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.843 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2022-11-27T17:42:42.846 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-11-27T17:42:42.847 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2022-11-27T17:42:42.847 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.847 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.847 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.848 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-11-27T17:42:42.849 INFO:teuthology.run_tasks:Running task install... 2022-11-27T17:42:42.858 DEBUG:teuthology.task.install:project ceph 2022-11-27T17:42:42.859 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}} 2022-11-27T17:42:42.859 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'} 2022-11-27T17:42:42.859 INFO:teuthology.task.install:Using flavor: default 2022-11-27T17:42:42.866 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-11-27T17:42:42.867 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-11-27T17:42:42.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-11-27T17:42:42.909 INFO:teuthology.orchestra.run.smithi099.stdout:RedHatEnterprise 2022-11-27T17:42:42.910 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:42:42.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-11-27T17:42:42.911 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'tag': None, 'wait_for_package': False} 2022-11-27T17:42:42.912 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-11-27T17:42:42.954 INFO:teuthology.orchestra.run.smithi119.stdout:RedHatEnterprise 2022-11-27T17:42:42.956 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:42:42.957 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'tag': None, 'wait_for_package': False} 2022-11-27T17:42:42.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-11-27T17:42:42.990 INFO:teuthology.orchestra.run.smithi099.stdout:RedHatEnterprise 2022-11-27T17:42:42.991 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:42:42.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-27T17:42:43.036 INFO:teuthology.orchestra.run.smithi119.stdout:RedHatEnterprise 2022-11-27T17:42:43.037 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:42:43.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-27T17:42:43.185 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/ 2022-11-27T17:42:43.186 INFO:teuthology.task.install.rpm:Package version is 18.0.0-594.g697b0d99 2022-11-27T17:42:43.250 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/ 2022-11-27T17:42:43.250 INFO:teuthology.task.install.rpm:Package version is 18.0.0-594.g697b0d99 2022-11-27T17:42:43.316 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-11-27T17:42:43.317 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:42:43.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-11-27T17:42:43.344 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-11-27T17:42:43.344 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/697b0d99d9a52df92b20c8e053dcd46c75f364a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-11-27T17:42:43.359 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/697b0d99d9a52df92b20c8e053dcd46c75f364a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-11-27T17:42:43.359 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:42:43.359 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-11-27T17:42:43.386 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-11-27T17:42:43.386 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/697b0d99d9a52df92b20c8e053dcd46c75f364a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-11-27T17:42:43.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-11-27T17:42:43.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-11-27T17:42:43.487 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-11-27T17:42:43.531 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-11-27T17:42:43.553 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2022-11-27T17:42:43.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2022-11-27T17:42:43.597 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2022-11-27T17:42:43.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2022-11-27T17:42:43.926 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2022-11-27T17:42:43.968 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2022-11-27T17:42:44.153 INFO:teuthology.orchestra.run.smithi119.stdout:54 files removed 2022-11-27T17:42:44.181 INFO:teuthology.orchestra.run.smithi099.stdout:54 files removed 2022-11-27T17:42:44.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-11-27T17:42:44.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-11-27T17:42:44.530 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2022-11-27T17:42:44.559 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2022-11-27T17:42:45.147 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 295 kB/s | 81 kB 00:00 2022-11-27T17:42:45.154 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 300 kB/s | 81 kB 00:00 2022-11-27T17:42:45.414 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2022-11-27T17:42:45.430 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2022-11-27T17:42:45.665 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 6.6 kB/s | 1.5 kB 00:00 2022-11-27T17:42:45.714 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 5.8 kB/s | 1.5 kB 00:00 2022-11-27T17:42:48.181 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-11-27T17:42:48.250 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-11-27T17:42:50.231 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 13 MB 00:02 2022-11-27T17:42:51.875 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 13 MB 00:03 2022-11-27T17:42:53.256 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 385 kB/s | 24 kB 00:00 2022-11-27T17:42:54.092 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 48 MB 00:00 2022-11-27T17:42:54.858 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 438 kB/s | 24 kB 00:00 2022-11-27T17:42:55.726 INFO:teuthology.orchestra.run.smithi119.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 66 MB/s | 53 MB 00:00 2022-11-27T17:43:02.509 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 67 MB/s | 53 MB 00:00 2022-11-27T17:43:03.786 INFO:teuthology.orchestra.run.smithi119.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 MB/s | 48 MB 00:00 2022-11-27T17:43:13.582 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-11-27T17:43:13.583 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-11-27T17:43:13.598 INFO:teuthology.orchestra.run.smithi099.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-11-27T17:43:13.600 INFO:teuthology.orchestra.run.smithi099.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-11-27T17:43:13.601 INFO:teuthology.orchestra.run.smithi099.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-11-27T17:43:13.602 INFO:teuthology.orchestra.run.smithi099.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-11-27T17:43:13.604 INFO:teuthology.orchestra.run.smithi099.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-11-27T17:43:13.608 INFO:teuthology.orchestra.run.smithi099.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-11-27T17:43:13.609 INFO:teuthology.orchestra.run.smithi099.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-11-27T17:43:13.739 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2022-11-27T17:43:13.751 INFO:teuthology.orchestra.run.smithi099.stdout:=========================================================================================================================== 2022-11-27T17:43:13.751 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2022-11-27T17:43:13.751 INFO:teuthology.orchestra.run.smithi099.stdout:=========================================================================================================================== 2022-11-27T17:43:13.752 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2022-11-27T17:43:13.752 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2022-11-27T17:43:13.752 INFO:teuthology.orchestra.run.smithi099.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-11-27T17:43:13.752 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.4 k 2022-11-27T17:43:13.753 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:18.0.0-594.g697b0d99.el8 ceph 5.2 M 2022-11-27T17:43:13.753 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 845 k 2022-11-27T17:43:13.753 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-594.g697b0d99.el8 ceph 165 k 2022-11-27T17:43:13.753 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:18.0.0-594.g697b0d99.el8 ceph 1.5 M 2022-11-27T17:43:13.754 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 118 k 2022-11-27T17:43:13.754 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 1.7 M 2022-11-27T17:43:13.754 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 7.4 M 2022-11-27T17:43:13.754 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 47 k 2022-11-27T17:43:13.755 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 18 M 2022-11-27T17:43:13.755 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:18.0.0-594.g697b0d99.el8 ceph 55 M 2022-11-27T17:43:13.755 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 263 k 2022-11-27T17:43:13.755 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 81 k 2022-11-27T17:43:13.756 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-mirror x86_64 2:18.0.0-594.g697b0d99.el8 ceph 257 k 2022-11-27T17:43:13.756 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-top noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 15 k 2022-11-27T17:43:13.756 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-11-27T17:43:13.756 INFO:teuthology.orchestra.run.smithi099.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-11-27T17:43:13.757 INFO:teuthology.orchestra.run.smithi099.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-11-27T17:43:13.757 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 30 k 2022-11-27T17:43:13.757 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 767 k 2022-11-27T17:43:13.757 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 139 k 2022-11-27T17:43:13.758 INFO:teuthology.orchestra.run.smithi099.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-11-27T17:43:13.758 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-11-27T17:43:13.758 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-11-27T17:43:13.758 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-11-27T17:43:13.758 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-11-27T17:43:13.759 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-11-27T17:43:13.759 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:18.0.0-594.g697b0d99.el8 ceph 190 k 2022-11-27T17:43:13.759 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:18.0.0-594.g697b0d99.el8 ceph 371 k 2022-11-27T17:43:13.759 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 357 k 2022-11-27T17:43:13.760 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 107 k 2022-11-27T17:43:13.760 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 94 k 2022-11-27T17:43:13.760 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.5 M 2022-11-27T17:43:13.760 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 175 k 2022-11-27T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-11-27T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2022-11-27T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2022-11-27T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-27T17:43:13.761 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-27T17:43:13.762 INFO:teuthology.orchestra.run.smithi099.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-11-27T17:43:13.762 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-11-27T17:43:13.762 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2022-11-27T17:43:13.762 INFO:teuthology.orchestra.run.smithi099.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-11-27T17:43:13.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 M 2022-11-27T17:43:13.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 23 k 2022-11-27T17:43:13.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:18.0.0-594.g697b0d99.el8 ceph 2.3 M 2022-11-27T17:43:13.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 234 k 2022-11-27T17:43:13.764 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.9 M 2022-11-27T17:43:13.764 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 17 M 2022-11-27T17:43:13.764 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 14 k 2022-11-27T17:43:13.764 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 k 2022-11-27T17:43:13.765 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-11-27T17:43:13.765 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-11-27T17:43:13.765 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-11-27T17:43:13.765 INFO:teuthology.orchestra.run.smithi099.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-11-27T17:43:13.765 INFO:teuthology.orchestra.run.smithi099.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-11-27T17:43:13.766 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:18.0.0-594.g697b0d99.el8 ceph 137 k 2022-11-27T17:43:13.766 INFO:teuthology.orchestra.run.smithi099.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-11-27T17:43:13.766 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-11-27T17:43:13.766 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-11-27T17:43:13.767 INFO:teuthology.orchestra.run.smithi099.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-11-27T17:43:13.767 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-11-27T17:43:13.767 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-11-27T17:43:13.767 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 474 k 2022-11-27T17:43:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-11-27T17:43:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.5 M 2022-11-27T17:43:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-11-27T17:43:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-11-27T17:43:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-11-27T17:43:13.769 INFO:teuthology.orchestra.run.smithi099.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-11-27T17:43:13.769 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-11-27T17:43:13.769 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-11-27T17:43:13.769 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-11-27T17:43:13.770 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-11-27T17:43:13.770 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-11-27T17:43:13.770 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-11-27T17:43:13.770 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-11-27T17:43:13.770 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-11-27T17:43:13.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-11-27T17:43:13.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-11-27T17:43:13.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-11-27T17:43:13.771 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-11-27T17:43:13.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 45 k 2022-11-27T17:43:13.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 109 k 2022-11-27T17:43:13.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-11-27T17:43:13.772 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-11-27T17:43:13.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-11-27T17:43:13.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-11-27T17:43:13.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-11-27T17:43:13.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-11-27T17:43:13.773 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-11-27T17:43:13.774 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-11-27T17:43:13.774 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-11-27T17:43:13.774 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-11-27T17:43:13.774 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-11-27T17:43:13.775 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-11-27T17:43:13.775 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-11-27T17:43:13.775 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-11-27T17:43:13.775 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-11-27T17:43:13.776 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-11-27T17:43:13.776 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-11-27T17:43:13.776 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-11-27T17:43:13.776 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-11-27T17:43:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-11-27T17:43:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-11-27T17:43:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-11-27T17:43:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-11-27T17:43:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-11-27T17:43:13.778 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-11-27T17:43:13.778 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-11-27T17:43:13.778 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-11-27T17:43:13.778 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-11-27T17:43:13.779 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-11-27T17:43:13.779 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-11-27T17:43:13.779 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-11-27T17:43:13.779 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-11-27T17:43:13.780 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-11-27T17:43:13.780 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-11-27T17:43:13.780 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-11-27T17:43:13.780 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-11-27T17:43:13.781 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-11-27T17:43:13.781 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-11-27T17:43:13.781 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-11-27T17:43:13.781 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-11-27T17:43:13.782 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-11-27T17:43:13.782 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-11-27T17:43:13.782 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-11-27T17:43:13.782 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-11-27T17:43:13.782 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-11-27T17:43:13.783 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-11-27T17:43:13.783 INFO:teuthology.orchestra.run.smithi099.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-11-27T17:43:13.783 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-11-27T17:43:13.783 INFO:teuthology.orchestra.run.smithi099.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-11-27T17:43:13.784 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling module streams: 2022-11-27T17:43:13.784 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql 10 2022-11-27T17:43:13.784 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:43:13.784 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2022-11-27T17:43:13.785 INFO:teuthology.orchestra.run.smithi099.stdout:=========================================================================================================================== 2022-11-27T17:43:13.785 INFO:teuthology.orchestra.run.smithi099.stdout:Install 128 Packages 2022-11-27T17:43:13.785 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 5 Packages 2022-11-27T17:43:13.785 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:43:13.786 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 219 M 2022-11-27T17:43:13.786 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2022-11-27T17:43:13.900 INFO:teuthology.orchestra.run.smithi099.stdout:(1/133): ceph-18.0.0-594.g697b0d99.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2022-11-27T17:43:14.092 INFO:teuthology.orchestra.run.smithi099.stdout:(2/133): ceph-fuse-18.0.0-594.g697b0d99.el8.x86 4.3 MB/s | 845 kB 00:00 2022-11-27T17:43:14.142 INFO:teuthology.orchestra.run.smithi099.stdout:(3/133): ceph-immutable-object-cache-18.0.0-594 3.2 MB/s | 165 kB 00:00 2022-11-27T17:43:14.501 INFO:teuthology.orchestra.run.smithi099.stdout:(4/133): ceph-mds-18.0.0-594.g697b0d99.el8.x86_ 6.3 MB/s | 2.3 MB 00:00 2022-11-27T17:43:14.576 INFO:teuthology.orchestra.run.smithi099.stdout:(5/133): ceph-base-18.0.0-594.g697b0d99.el8.x86 6.4 MB/s | 5.2 MB 00:00 2022-11-27T17:43:14.776 INFO:teuthology.orchestra.run.smithi099.stdout:(6/133): ceph-mgr-18.0.0-594.g697b0d99.el8.x86_ 5.6 MB/s | 1.5 MB 00:00 2022-11-27T17:43:14.960 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-11-27T17:43:14.961 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-11-27T17:43:14.976 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-11-27T17:43:14.977 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-11-27T17:43:14.978 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-11-27T17:43:14.979 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-11-27T17:43:14.981 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-11-27T17:43:14.984 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-11-27T17:43:14.986 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-11-27T17:43:15.052 INFO:teuthology.orchestra.run.smithi099.stdout:(7/133): ceph-mon-18.0.0-594.g697b0d99.el8.x86_ 8.1 MB/s | 3.9 MB 00:00 2022-11-27T17:43:15.121 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2022-11-27T17:43:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:=========================================================================================================================== 2022-11-27T17:43:15.133 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2022-11-27T17:43:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:=========================================================================================================================== 2022-11-27T17:43:15.134 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2022-11-27T17:43:15.134 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2022-11-27T17:43:15.134 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-11-27T17:43:15.134 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.4 k 2022-11-27T17:43:15.135 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:18.0.0-594.g697b0d99.el8 ceph 5.2 M 2022-11-27T17:43:15.135 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 845 k 2022-11-27T17:43:15.135 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-594.g697b0d99.el8 ceph 165 k 2022-11-27T17:43:15.135 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:18.0.0-594.g697b0d99.el8 ceph 1.5 M 2022-11-27T17:43:15.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 118 k 2022-11-27T17:43:15.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 1.7 M 2022-11-27T17:43:15.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 7.4 M 2022-11-27T17:43:15.136 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 47 k 2022-11-27T17:43:15.137 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 18 M 2022-11-27T17:43:15.137 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:18.0.0-594.g697b0d99.el8 ceph 55 M 2022-11-27T17:43:15.137 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 263 k 2022-11-27T17:43:15.137 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 81 k 2022-11-27T17:43:15.138 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:18.0.0-594.g697b0d99.el8 ceph 257 k 2022-11-27T17:43:15.138 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 15 k 2022-11-27T17:43:15.138 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-11-27T17:43:15.138 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-11-27T17:43:15.139 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-11-27T17:43:15.139 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 30 k 2022-11-27T17:43:15.139 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 767 k 2022-11-27T17:43:15.139 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:18.0.0-594.g697b0d99.el8 ceph 139 k 2022-11-27T17:43:15.140 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-11-27T17:43:15.140 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-11-27T17:43:15.140 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-11-27T17:43:15.140 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-11-27T17:43:15.140 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-11-27T17:43:15.141 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-11-27T17:43:15.141 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:18.0.0-594.g697b0d99.el8 ceph 190 k 2022-11-27T17:43:15.141 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:18.0.0-594.g697b0d99.el8 ceph 371 k 2022-11-27T17:43:15.141 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 357 k 2022-11-27T17:43:15.142 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:18.0.0-594.g697b0d99.el8 ceph 107 k 2022-11-27T17:43:15.142 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 94 k 2022-11-27T17:43:15.142 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.5 M 2022-11-27T17:43:15.142 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 175 k 2022-11-27T17:43:15.142 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-11-27T17:43:15.143 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2022-11-27T17:43:15.143 INFO:teuthology.orchestra.run.smithi119.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2022-11-27T17:43:15.143 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-27T17:43:15.143 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.6 M 2022-11-27T17:43:15.144 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-11-27T17:43:15.144 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-11-27T17:43:15.144 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2022-11-27T17:43:15.144 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-11-27T17:43:15.144 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 M 2022-11-27T17:43:15.145 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 23 k 2022-11-27T17:43:15.145 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:18.0.0-594.g697b0d99.el8 ceph 2.3 M 2022-11-27T17:43:15.145 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 234 k 2022-11-27T17:43:15.145 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:18.0.0-594.g697b0d99.el8 ceph 3.9 M 2022-11-27T17:43:15.145 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:18.0.0-594.g697b0d99.el8 ceph 17 M 2022-11-27T17:43:15.146 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:18.0.0-594.g697b0d99.el8 ceph-noarch 14 k 2022-11-27T17:43:15.146 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:18.0.0-594.g697b0d99.el8 ceph 24 k 2022-11-27T17:43:15.146 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-11-27T17:43:15.146 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-11-27T17:43:15.146 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-11-27T17:43:15.147 INFO:teuthology.orchestra.run.smithi119.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-11-27T17:43:15.147 INFO:teuthology.orchestra.run.smithi119.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-11-27T17:43:15.147 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:18.0.0-594.g697b0d99.el8 ceph 137 k 2022-11-27T17:43:15.147 INFO:teuthology.orchestra.run.smithi119.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-11-27T17:43:15.148 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-11-27T17:43:15.148 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-11-27T17:43:15.148 INFO:teuthology.orchestra.run.smithi119.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-11-27T17:43:15.148 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-11-27T17:43:15.149 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-11-27T17:43:15.149 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 474 k 2022-11-27T17:43:15.149 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-11-27T17:43:15.150 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:18.0.0-594.g697b0d99.el8 ceph 6.5 M 2022-11-27T17:43:15.150 INFO:teuthology.orchestra.run.smithi119.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-11-27T17:43:15.150 INFO:teuthology.orchestra.run.smithi119.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-11-27T17:43:15.150 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-11-27T17:43:15.150 INFO:teuthology.orchestra.run.smithi119.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-11-27T17:43:15.151 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-11-27T17:43:15.151 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-11-27T17:43:15.151 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-11-27T17:43:15.151 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-11-27T17:43:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-11-27T17:43:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-11-27T17:43:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-11-27T17:43:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-11-27T17:43:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-11-27T17:43:15.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-11-27T17:43:15.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-11-27T17:43:15.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-11-27T17:43:15.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:18.0.0-594.g697b0d99.el8 ceph 45 k 2022-11-27T17:43:15.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:18.0.0-594.g697b0d99.el8 ceph 109 k 2022-11-27T17:43:15.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-11-27T17:43:15.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-11-27T17:43:15.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-11-27T17:43:15.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-11-27T17:43:15.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-11-27T17:43:15.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-11-27T17:43:15.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-11-27T17:43:15.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-11-27T17:43:15.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-11-27T17:43:15.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-11-27T17:43:15.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-11-27T17:43:15.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-11-27T17:43:15.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-11-27T17:43:15.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-11-27T17:43:15.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-11-27T17:43:15.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-11-27T17:43:15.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-11-27T17:43:15.158 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-11-27T17:43:15.158 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-11-27T17:43:15.158 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-11-27T17:43:15.158 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-11-27T17:43:15.159 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-11-27T17:43:15.159 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-11-27T17:43:15.159 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-11-27T17:43:15.159 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-11-27T17:43:15.159 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-11-27T17:43:15.160 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-11-27T17:43:15.160 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-11-27T17:43:15.160 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-11-27T17:43:15.160 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-11-27T17:43:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-11-27T17:43:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-11-27T17:43:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-11-27T17:43:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-11-27T17:43:15.162 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-11-27T17:43:15.162 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-11-27T17:43:15.162 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-11-27T17:43:15.162 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-11-27T17:43:15.162 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-11-27T17:43:15.163 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-11-27T17:43:15.163 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-11-27T17:43:15.163 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-11-27T17:43:15.164 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-11-27T17:43:15.164 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-11-27T17:43:15.164 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-11-27T17:43:15.164 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-11-27T17:43:15.164 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-11-27T17:43:15.165 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-11-27T17:43:15.165 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-11-27T17:43:15.165 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling module streams: 2022-11-27T17:43:15.165 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql 10 2022-11-27T17:43:15.166 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:43:15.166 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2022-11-27T17:43:15.166 INFO:teuthology.orchestra.run.smithi119.stdout:=========================================================================================================================== 2022-11-27T17:43:15.166 INFO:teuthology.orchestra.run.smithi119.stdout:Install 128 Packages 2022-11-27T17:43:15.167 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 5 Packages 2022-11-27T17:43:15.167 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:43:15.168 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 219 M 2022-11-27T17:43:15.168 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2022-11-27T17:43:15.494 INFO:teuthology.orchestra.run.smithi119.stdout:(1/133): ceph-18.0.0-594.g697b0d99.el8.x86_64.r 19 kB/s | 6.4 kB 00:00 2022-11-27T17:43:16.161 INFO:teuthology.orchestra.run.smithi119.stdout:(2/133): ceph-fuse-18.0.0-594.g697b0d99.el8.x86 1.2 MB/s | 845 kB 00:00 2022-11-27T17:43:16.269 INFO:teuthology.orchestra.run.smithi119.stdout:(3/133): ceph-immutable-object-cache-18.0.0-594 1.5 MB/s | 165 kB 00:00 2022-11-27T17:43:16.335 INFO:teuthology.orchestra.run.smithi099.stdout:(8/133): ceph-common-18.0.0-594.g697b0d99.el8.x 9.5 MB/s | 24 MB 00:02 2022-11-27T17:43:16.419 INFO:teuthology.orchestra.run.smithi099.stdout:(9/133): ceph-selinux-18.0.0-594.g697b0d99.el8. 292 kB/s | 24 kB 00:00 2022-11-27T17:43:16.728 INFO:teuthology.orchestra.run.smithi119.stdout:(4/133): ceph-base-18.0.0-594.g697b0d99.el8.x86 3.3 MB/s | 5.2 MB 00:01 2022-11-27T17:43:17.078 INFO:teuthology.orchestra.run.smithi119.stdout:(5/133): ceph-mgr-18.0.0-594.g697b0d99.el8.x86_ 4.4 MB/s | 1.5 MB 00:00 2022-11-27T17:43:17.253 INFO:teuthology.orchestra.run.smithi119.stdout:(6/133): ceph-mds-18.0.0-594.g697b0d99.el8.x86_ 2.3 MB/s | 2.3 MB 00:00 2022-11-27T17:43:17.277 INFO:teuthology.orchestra.run.smithi099.stdout:(10/133): ceph-osd-18.0.0-594.g697b0d99.el8.x86 6.7 MB/s | 17 MB 00:02 2022-11-27T17:43:17.369 INFO:teuthology.orchestra.run.smithi099.stdout:(11/133): cephfs-mirror-18.0.0-594.g697b0d99.el 2.7 MB/s | 257 kB 00:00 2022-11-27T17:43:17.444 INFO:teuthology.orchestra.run.smithi099.stdout:(12/133): libcephfs-devel-18.0.0-594.g697b0d99. 408 kB/s | 30 kB 00:00 2022-11-27T17:43:17.678 INFO:teuthology.orchestra.run.smithi099.stdout:(13/133): libcephfs2-18.0.0-594.g697b0d99.el8.x 3.2 MB/s | 767 kB 00:00 2022-11-27T17:43:17.811 INFO:teuthology.orchestra.run.smithi099.stdout:(14/133): libcephsqlite-18.0.0-594.g697b0d99.el 1.0 MB/s | 137 kB 00:00 2022-11-27T17:43:17.920 INFO:teuthology.orchestra.run.smithi099.stdout:(15/133): librados-devel-18.0.0-594.g697b0d99.e 1.2 MB/s | 139 kB 00:00 2022-11-27T17:43:18.088 INFO:teuthology.orchestra.run.smithi099.stdout:(16/133): ceph-radosgw-18.0.0-594.g697b0d99.el8 6.0 MB/s | 18 MB 00:03 2022-11-27T17:43:18.113 INFO:teuthology.orchestra.run.smithi099.stdout:(17/133): libradosstriper1-18.0.0-594.g697b0d99 2.4 MB/s | 474 kB 00:00 2022-11-27T17:43:18.205 INFO:teuthology.orchestra.run.smithi099.stdout:(18/133): python3-ceph-argparse-18.0.0-594.g697 486 kB/s | 45 kB 00:00 2022-11-27T17:43:18.289 INFO:teuthology.orchestra.run.smithi099.stdout:(19/133): python3-ceph-common-18.0.0-594.g697b0 1.3 MB/s | 109 kB 00:00 2022-11-27T17:43:18.295 INFO:teuthology.orchestra.run.smithi119.stdout:(7/133): ceph-mon-18.0.0-594.g697b0d99.el8.x86_ 3.2 MB/s | 3.9 MB 00:01 2022-11-27T17:43:18.372 INFO:teuthology.orchestra.run.smithi099.stdout:(20/133): python3-cephfs-18.0.0-594.g697b0d99.e 2.2 MB/s | 190 kB 00:00 2022-11-27T17:43:18.481 INFO:teuthology.orchestra.run.smithi099.stdout:(21/133): python3-rados-18.0.0-594.g697b0d99.el 3.4 MB/s | 371 kB 00:00 2022-11-27T17:43:18.589 INFO:teuthology.orchestra.run.smithi099.stdout:(22/133): python3-rbd-18.0.0-594.g697b0d99.el8. 3.2 MB/s | 357 kB 00:00 2022-11-27T17:43:18.673 INFO:teuthology.orchestra.run.smithi099.stdout:(23/133): python3-rgw-18.0.0-594.g697b0d99.el8. 1.3 MB/s | 107 kB 00:00 2022-11-27T17:43:18.757 INFO:teuthology.orchestra.run.smithi099.stdout:(24/133): rbd-fuse-18.0.0-594.g697b0d99.el8.x86 1.1 MB/s | 94 kB 00:00 2022-11-27T17:43:19.540 INFO:teuthology.orchestra.run.smithi099.stdout:(25/133): rbd-mirror-18.0.0-594.g697b0d99.el8.x 4.4 MB/s | 3.5 MB 00:00 2022-11-27T17:43:19.656 INFO:teuthology.orchestra.run.smithi099.stdout:(26/133): rbd-nbd-18.0.0-594.g697b0d99.el8.x86_ 1.5 MB/s | 175 kB 00:00 2022-11-27T17:43:19.749 INFO:teuthology.orchestra.run.smithi099.stdout:(27/133): ceph-grafana-dashboards-18.0.0-594.g6 247 kB/s | 23 kB 00:00 2022-11-27T17:43:19.824 INFO:teuthology.orchestra.run.smithi099.stdout:(28/133): ceph-mgr-cephadm-18.0.0-594.g697b0d99 1.5 MB/s | 118 kB 00:00 2022-11-27T17:43:20.035 INFO:teuthology.orchestra.run.smithi099.stdout:(29/133): librgw2-18.0.0-594.g697b0d99.el8.x86_ 3.3 MB/s | 6.5 MB 00:01 2022-11-27T17:43:20.308 INFO:teuthology.orchestra.run.smithi099.stdout:(30/133): ceph-mgr-dashboard-18.0.0-594.g697b0d 3.4 MB/s | 1.7 MB 00:00 2022-11-27T17:43:20.425 INFO:teuthology.orchestra.run.smithi099.stdout:(31/133): ceph-mgr-modules-core-18.0.0-594.g697 2.0 MB/s | 234 kB 00:00 2022-11-27T17:43:20.524 INFO:teuthology.orchestra.run.smithi099.stdout:(32/133): ceph-mgr-rook-18.0.0-594.g697b0d99.el 512 kB/s | 47 kB 00:00 2022-11-27T17:43:20.609 INFO:teuthology.orchestra.run.smithi099.stdout:(33/133): ceph-prometheus-alerts-18.0.0-594.g69 157 kB/s | 14 kB 00:00 2022-11-27T17:43:20.710 INFO:teuthology.orchestra.run.smithi099.stdout:(34/133): ceph-volume-18.0.0-594.g697b0d99.el8. 2.6 MB/s | 263 kB 00:00 2022-11-27T17:43:20.810 INFO:teuthology.orchestra.run.smithi099.stdout:(35/133): cephadm-18.0.0-594.g697b0d99.el8.noar 812 kB/s | 81 kB 00:00 2022-11-27T17:43:20.902 INFO:teuthology.orchestra.run.smithi099.stdout:(36/133): cephfs-top-18.0.0-594.g697b0d99.el8.n 168 kB/s | 15 kB 00:00 2022-11-27T17:43:21.094 INFO:teuthology.orchestra.run.smithi099.stdout:(37/133): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2022-11-27T17:43:21.169 INFO:teuthology.orchestra.run.smithi099.stdout:(38/133): liboath-2.6.2-3.el8.x86_64.rpm 785 kB/s | 59 kB 00:00 2022-11-27T17:43:21.219 INFO:teuthology.orchestra.run.smithi099.stdout:(39/133): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-11-27T17:43:21.303 INFO:teuthology.orchestra.run.smithi099.stdout:(40/133): python3-asyncssh-2.7.0-2.el8.noarch.r 5.6 MB/s | 476 kB 00:00 2022-11-27T17:43:21.362 INFO:teuthology.orchestra.run.smithi099.stdout:(41/133): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 742 kB/s | 44 kB 00:00 2022-11-27T17:43:21.412 INFO:teuthology.orchestra.run.smithi099.stdout:(42/133): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2022-11-27T17:43:21.454 INFO:teuthology.orchestra.run.smithi099.stdout:(43/133): python3-cachetools-3.1.1-4.el8.noarch 792 kB/s | 33 kB 00:00 2022-11-27T17:43:21.529 INFO:teuthology.orchestra.run.smithi099.stdout:(44/133): python3-certifi-2018.10.15-7.el8.noar 217 kB/s | 16 kB 00:00 2022-11-27T17:43:21.579 INFO:teuthology.orchestra.run.smithi099.stdout:(45/133): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2022-11-27T17:43:21.705 INFO:teuthology.orchestra.run.smithi099.stdout:(46/133): python3-cherrypy-18.4.0-1.el8.noarch. 3.0 MB/s | 384 kB 00:00 2022-11-27T17:43:21.755 INFO:teuthology.orchestra.run.smithi099.stdout:(47/133): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2022-11-27T17:43:21.797 INFO:teuthology.orchestra.run.smithi099.stdout:(48/133): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2022-11-27T17:43:21.878 INFO:teuthology.orchestra.run.smithi099.stdout:(49/133): python3-jaraco-functools-2.0-4.el8.no 219 kB/s | 18 kB 00:00 2022-11-27T17:43:22.023 INFO:teuthology.orchestra.run.smithi099.stdout:(50/133): python3-kubernetes-11.0.0-6.el8.noarc 8.0 MB/s | 1.2 MB 00:00 2022-11-27T17:43:22.073 INFO:teuthology.orchestra.run.smithi099.stdout:(51/133): python3-logutils-0.3.5-11.el8.noarch. 979 kB/s | 49 kB 00:00 2022-11-27T17:43:22.124 INFO:teuthology.orchestra.run.smithi099.stdout:(52/133): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2022-11-27T17:43:22.174 INFO:teuthology.orchestra.run.smithi099.stdout:(53/133): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2022-11-27T17:43:22.265 INFO:teuthology.orchestra.run.smithi099.stdout:(54/133): python3-pecan-1.3.2-9.el8.noarch.rpm 3.1 MB/s | 283 kB 00:00 2022-11-27T17:43:22.316 INFO:teuthology.orchestra.run.smithi099.stdout:(55/133): python3-portend-2.6-1.el8.noarch.rpm 305 kB/s | 16 kB 00:00 2022-11-27T17:43:22.367 INFO:teuthology.orchestra.run.smithi099.stdout:(56/133): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2022-11-27T17:43:22.417 INFO:teuthology.orchestra.run.smithi099.stdout:(57/133): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2022-11-27T17:43:22.467 INFO:teuthology.orchestra.run.smithi099.stdout:(58/133): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2022-11-27T17:43:22.518 INFO:teuthology.orchestra.run.smithi099.stdout:(59/133): python3-simplegeneric-0.8.1-17.el8.no 373 kB/s | 19 kB 00:00 2022-11-27T17:43:22.568 INFO:teuthology.orchestra.run.smithi099.stdout:(60/133): python3-singledispatch-3.4.0.3-18.el8 479 kB/s | 24 kB 00:00 2022-11-27T17:43:22.652 INFO:teuthology.orchestra.run.smithi099.stdout:(61/133): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2022-11-27T17:43:22.710 INFO:teuthology.orchestra.run.smithi099.stdout:(62/133): python3-trustme-0.6.0-4.el8.noarch.rp 474 kB/s | 27 kB 00:00 2022-11-27T17:43:22.811 INFO:teuthology.orchestra.run.smithi099.stdout:(63/133): ceph-mgr-diskprediction-local-18.0.0- 2.7 MB/s | 7.4 MB 00:02 2022-11-27T17:43:22.836 INFO:teuthology.orchestra.run.smithi099.stdout:(64/133): python3-waitress-1.2.1-2.el8.1.noarch 1.9 MB/s | 241 kB 00:00 2022-11-27T17:43:22.887 INFO:teuthology.orchestra.run.smithi099.stdout:(65/133): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-11-27T17:43:22.937 INFO:teuthology.orchestra.run.smithi099.stdout:(66/133): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2022-11-27T17:43:23.013 INFO:teuthology.orchestra.run.smithi099.stdout:(67/133): python3-zc-lockfile-2.0-2.el8.noarch. 300 kB/s | 23 kB 00:00 2022-11-27T17:43:23.045 INFO:teuthology.orchestra.run.smithi099.stdout:(68/133): python3-webob-1.8.5-1.el8.1.noarch.rp 1.0 MB/s | 251 kB 00:00 2022-11-27T17:43:23.088 INFO:teuthology.orchestra.run.smithi099.stdout:(69/133): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2022-11-27T17:43:23.146 INFO:teuthology.orchestra.run.smithi099.stdout:(70/133): postgresql-contrib-10.21-2.module+el8 14 MB/s | 808 kB 00:00 2022-11-27T17:43:23.197 INFO:teuthology.orchestra.run.smithi099.stdout:(71/133): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2022-11-27T17:43:23.247 INFO:teuthology.orchestra.run.smithi099.stdout:(72/133): postgresql-10.21-2.module+el8.6.0+153 30 MB/s | 1.5 MB 00:00 2022-11-27T17:43:23.281 INFO:teuthology.orchestra.run.smithi099.stdout:(73/133): thrift-0.13.0-2.el8.x86_64.rpm 6.5 MB/s | 1.7 MB 00:00 2022-11-27T17:43:23.414 INFO:teuthology.orchestra.run.smithi099.stdout:(74/133): python3-numpy-1.14.3-10.el8.x86_64.rp 22 MB/s | 3.7 MB 00:00 2022-11-27T17:43:23.490 INFO:teuthology.orchestra.run.smithi099.stdout:(75/133): postgresql-server-10.21-2.module+el8. 24 MB/s | 5.1 MB 00:00 2022-11-27T17:43:23.515 INFO:teuthology.orchestra.run.smithi099.stdout:(76/133): python3-werkzeug-0.12.2-4.el8.noarch. 4.5 MB/s | 457 kB 00:00 2022-11-27T17:43:23.540 INFO:teuthology.orchestra.run.smithi099.stdout:(77/133): python3-markupsafe-0.23-19.el8.x86_64 776 kB/s | 39 kB 00:00 2022-11-27T17:43:23.557 INFO:teuthology.orchestra.run.smithi099.stdout:(78/133): python3-prettytable-0.7.2-14.el8.noar 1.1 MB/s | 44 kB 00:00 2022-11-27T17:43:23.599 INFO:teuthology.orchestra.run.smithi099.stdout:(79/133): librdkafka-0.11.4-3.el8.x86_64.rpm 5.9 MB/s | 354 kB 00:00 2022-11-27T17:43:23.641 INFO:teuthology.orchestra.run.smithi099.stdout:(80/133): python3-jinja2-2.10.1-3.el8.noarch.rp 6.3 MB/s | 538 kB 00:00 2022-11-27T17:43:23.683 INFO:teuthology.orchestra.run.smithi099.stdout:(81/133): libtool-2.4.6-25.el8.x86_64.rpm 8.3 MB/s | 709 kB 00:00 2022-11-27T17:43:23.709 INFO:teuthology.orchestra.run.smithi099.stdout:(82/133): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.3 MB/s | 225 kB 00:00 2022-11-27T17:43:23.712 INFO:teuthology.orchestra.run.smithi119.stdout:(8/133): ceph-common-18.0.0-594.g697b0d99.el8.x 2.8 MB/s | 24 MB 00:08 2022-11-27T17:43:23.779 INFO:teuthology.orchestra.run.smithi119.stdout:(9/133): ceph-selinux-18.0.0-594.g697b0d99.el8. 367 kB/s | 24 kB 00:00 2022-11-27T17:43:24.079 INFO:teuthology.orchestra.run.smithi119.stdout:(10/133): ceph-osd-18.0.0-594.g697b0d99.el8.x86 2.5 MB/s | 17 MB 00:06 2022-11-27T17:43:24.200 INFO:teuthology.orchestra.run.smithi099.stdout:(83/133): ceph-test-18.0.0-594.g697b0d99.el8.x8 7.0 MB/s | 55 MB 00:07 2022-11-27T17:43:24.234 INFO:teuthology.orchestra.run.smithi099.stdout:(84/133): python3-mako-1.0.6-13.el8.noarch.rpm 299 kB/s | 157 kB 00:00 2022-11-27T17:43:24.276 INFO:teuthology.orchestra.run.smithi099.stdout:(85/133): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2022-11-27T17:43:24.288 INFO:teuthology.orchestra.run.smithi119.stdout:(11/133): cephfs-mirror-18.0.0-594.g697b0d99.el 1.2 MB/s | 257 kB 00:00 2022-11-27T17:43:24.318 INFO:teuthology.orchestra.run.smithi099.stdout:(86/133): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2022-11-27T17:43:24.363 INFO:teuthology.orchestra.run.smithi119.stdout:(12/133): libcephfs-devel-18.0.0-594.g697b0d99. 408 kB/s | 30 kB 00:00 2022-11-27T17:43:24.385 INFO:teuthology.orchestra.run.smithi099.stdout:(87/133): python3-babel-2.5.1-7.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2022-11-27T17:43:24.460 INFO:teuthology.orchestra.run.smithi099.stdout:(88/133): openblas-threads-0.3.15-3.el8.x86_64. 6.3 MB/s | 4.9 MB 00:00 2022-11-27T17:43:24.493 INFO:teuthology.orchestra.run.smithi099.stdout:(89/133): flex-2.6.1-9.el8.x86_64.rpm 1.8 MB/s | 320 kB 00:00 2022-11-27T17:43:24.527 INFO:teuthology.orchestra.run.smithi099.stdout:(90/133): jq-1.6-3.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2022-11-27T17:43:24.611 INFO:teuthology.orchestra.run.smithi099.stdout:(91/133): automake-1.16.1-7.el8.noarch.rpm 8.4 MB/s | 713 kB 00:00 2022-11-27T17:43:24.671 INFO:teuthology.orchestra.run.smithi119.stdout:(13/133): libcephfs2-18.0.0-594.g697b0d99.el8.x 2.4 MB/s | 767 kB 00:00 2022-11-27T17:43:24.680 INFO:teuthology.orchestra.run.smithi099.stdout:(92/133): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2022-11-27T17:43:24.714 INFO:teuthology.orchestra.run.smithi099.stdout:(93/133): python3-pytz-2017.2-9.el8.noarch.rpm 530 kB/s | 54 kB 00:00 2022-11-27T17:43:24.739 INFO:teuthology.orchestra.run.smithi099.stdout:(94/133): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.2 MB/s | 126 kB 00:00 2022-11-27T17:43:24.755 INFO:teuthology.orchestra.run.smithi119.stdout:(14/133): libcephsqlite-18.0.0-594.g697b0d99.el 1.6 MB/s | 137 kB 00:00 2022-11-27T17:43:24.781 INFO:teuthology.orchestra.run.smithi099.stdout:(95/133): python3-jsonpointer-1.10-11.el8.noarc 300 kB/s | 20 kB 00:00 2022-11-27T17:43:24.806 INFO:teuthology.orchestra.run.smithi099.stdout:(96/133): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2022-11-27T17:43:24.839 INFO:teuthology.orchestra.run.smithi119.stdout:(15/133): librados-devel-18.0.0-594.g697b0d99.e 1.6 MB/s | 139 kB 00:00 2022-11-27T17:43:24.848 INFO:teuthology.orchestra.run.smithi099.stdout:(97/133): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2022-11-27T17:43:24.873 INFO:teuthology.orchestra.run.smithi099.stdout:(98/133): oniguruma-6.8.2-2.el8.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2022-11-27T17:43:24.907 INFO:teuthology.orchestra.run.smithi099.stdout:(99/133): python3-jsonpatch-1.21-2.el8.noarch.r 454 kB/s | 27 kB 00:00 2022-11-27T17:43:24.949 INFO:teuthology.orchestra.run.smithi099.stdout:(100/133): uuid-1.6.2-43.el8.x86_64.rpm 864 kB/s | 64 kB 00:00 2022-11-27T17:43:24.991 INFO:teuthology.orchestra.run.smithi099.stdout:(101/133): python3-pyasn1-modules-0.3.7-6.el8.n 1.3 MB/s | 110 kB 00:00 2022-11-27T17:43:25.025 INFO:teuthology.orchestra.run.smithi099.stdout:(102/133): pcre2-devel-10.32-2.el8.x86_64.rpm 7.8 MB/s | 605 kB 00:00 2022-11-27T17:43:25.047 INFO:teuthology.orchestra.run.smithi119.stdout:(16/133): libradosstriper1-18.0.0-594.g697b0d99 2.2 MB/s | 474 kB 00:00 2022-11-27T17:43:25.167 INFO:teuthology.orchestra.run.smithi099.stdout:(103/133): python3-scipy-1.0.0-21.module+el8.5. 20 MB/s | 14 MB 00:00 2022-11-27T17:43:25.192 INFO:teuthology.orchestra.run.smithi099.stdout:(104/133): fuse-2.9.7-15.el8.x86_64.rpm 413 kB/s | 83 kB 00:00 2022-11-27T17:43:25.209 INFO:teuthology.orchestra.run.smithi099.stdout:(105/133): libsepol-devel-2.9-3.el8.x86_64.rpm 476 kB/s | 87 kB 00:00 2022-11-27T17:43:25.251 INFO:teuthology.orchestra.run.smithi099.stdout:(106/133): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2022-11-27T17:43:25.276 INFO:teuthology.orchestra.run.smithi099.stdout:(107/133): libverto-devel-0.3.0-5.el8.x86_64.rp 272 kB/s | 18 kB 00:00 2022-11-27T17:43:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:(108/133): libselinux-devel-2.9-5.el8.x86_64.rp 1.5 MB/s | 200 kB 00:00 2022-11-27T17:43:25.368 INFO:teuthology.orchestra.run.smithi099.stdout:(109/133): librabbitmq-0.9.0-3.el8.x86_64.rpm 399 kB/s | 47 kB 00:00 2022-11-27T17:43:25.385 INFO:teuthology.orchestra.run.smithi099.stdout:(110/133): python3-oauthlib-2.1.0-1.el8.noarch. 2.7 MB/s | 155 kB 00:00 2022-11-27T17:43:25.419 INFO:teuthology.orchestra.run.smithi099.stdout:(111/133): python3-cryptography-3.2.1-5.el8.x86 3.8 MB/s | 559 kB 00:00 2022-11-27T17:43:25.453 INFO:teuthology.orchestra.run.smithi099.stdout:(112/133): python3-pycparser-2.14-14.el8.noarch 1.3 MB/s | 109 kB 00:00 2022-11-27T17:43:25.486 INFO:teuthology.orchestra.run.smithi099.stdout:(113/133): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.5 MB/s | 560 kB 00:00 2022-11-27T17:43:25.511 INFO:teuthology.orchestra.run.smithi099.stdout:(114/133): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.9 MB/s | 171 kB 00:00 2022-11-27T17:43:25.545 INFO:teuthology.orchestra.run.smithi099.stdout:(115/133): python3-jwt-1.6.1-2.el8.noarch.rpm 468 kB/s | 43 kB 00:00 2022-11-27T17:43:25.556 INFO:teuthology.orchestra.run.smithi119.stdout:(17/133): ceph-radosgw-18.0.0-594.g697b0d99.el8 2.5 MB/s | 18 MB 00:07 2022-11-27T17:43:25.570 INFO:teuthology.orchestra.run.smithi099.stdout:(116/133): zlib-devel-1.2.11-18.el8_5.x86_64.rp 698 kB/s | 58 kB 00:00 2022-11-27T17:43:25.604 INFO:teuthology.orchestra.run.smithi099.stdout:(117/133): elfutils-libelf-devel-0.186-1.el8.x8 664 kB/s | 60 kB 00:00 2022-11-27T17:43:25.629 INFO:teuthology.orchestra.run.smithi099.stdout:(118/133): libaio-devel-0.3.112-1.el8.x86_64.rp 326 kB/s | 19 kB 00:00 2022-11-27T17:43:25.651 INFO:teuthology.orchestra.run.smithi119.stdout:(18/133): python3-ceph-argparse-18.0.0-594.g697 470 kB/s | 45 kB 00:00 2022-11-27T17:43:25.655 INFO:teuthology.orchestra.run.smithi099.stdout:(119/133): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2022-11-27T17:43:25.714 INFO:teuthology.orchestra.run.smithi099.stdout:(120/133): libuuid-devel-2.32.1-35.el8.x86_64.r 901 kB/s | 98 kB 00:00 2022-11-27T17:43:25.731 INFO:teuthology.orchestra.run.smithi119.stdout:(19/133): python3-ceph-common-18.0.0-594.g697b0 1.3 MB/s | 109 kB 00:00 2022-11-27T17:43:25.739 INFO:teuthology.orchestra.run.smithi099.stdout:(121/133): python3-requests-oauthlib-1.0.0-1.el 517 kB/s | 43 kB 00:00 2022-11-27T17:43:25.764 INFO:teuthology.orchestra.run.smithi099.stdout:(122/133): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2022-11-27T17:43:25.798 INFO:teuthology.orchestra.run.smithi099.stdout:(123/133): xfsprogs-devel-5.0.0-10.el8.x86_64.r 864 kB/s | 72 kB 00:00 2022-11-27T17:43:25.823 INFO:teuthology.orchestra.run.smithi099.stdout:(124/133): libcom_err-devel-1.45.6-4.el8.x86_64 461 kB/s | 39 kB 00:00 2022-11-27T17:43:25.824 INFO:teuthology.orchestra.run.smithi119.stdout:(20/133): python3-cephfs-18.0.0-594.g697b0d99.e 2.0 MB/s | 190 kB 00:00 2022-11-27T17:43:25.898 INFO:teuthology.orchestra.run.smithi099.stdout:(125/133): openssl-devel-1.1.1k-7.el8_6.x86_64. 18 MB/s | 2.3 MB 00:00 2022-11-27T17:43:25.924 INFO:teuthology.orchestra.run.smithi099.stdout:(126/133): keyutils-libs-devel-1.5.10-9.el8.x86 383 kB/s | 48 kB 00:00 2022-11-27T17:43:25.957 INFO:teuthology.orchestra.run.smithi119.stdout:(21/133): python3-rados-18.0.0-594.g697b0d99.el 2.7 MB/s | 371 kB 00:00 2022-11-27T17:43:25.958 INFO:teuthology.orchestra.run.smithi099.stdout:(127/133): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.7 MB/s | 644 kB 00:00 2022-11-27T17:43:25.991 INFO:teuthology.orchestra.run.smithi099.stdout:(128/133): NetworkManager-initscripts-updown-1. 1.5 MB/s | 138 kB 00:00 2022-11-27T17:43:26.066 INFO:teuthology.orchestra.run.smithi099.stdout:(129/133): openssl-1.1.1k-7.el8_6.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2022-11-27T17:43:26.107 INFO:teuthology.orchestra.run.smithi119.stdout:(22/133): python3-rbd-18.0.0-594.g697b0d99.el8. 2.3 MB/s | 357 kB 00:00 2022-11-27T17:43:26.133 INFO:teuthology.orchestra.run.smithi099.stdout:(130/133): openssl-libs-1.1.1k-7.el8_6.x86_64.r 22 MB/s | 1.5 MB 00:00 2022-11-27T17:43:26.183 INFO:teuthology.orchestra.run.smithi099.stdout:(131/133): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2022-11-27T17:43:26.208 INFO:teuthology.orchestra.run.smithi119.stdout:(23/133): python3-rgw-18.0.0-594.g697b0d99.el8. 1.0 MB/s | 107 kB 00:00 2022-11-27T17:43:26.308 INFO:teuthology.orchestra.run.smithi119.stdout:(24/133): rbd-fuse-18.0.0-594.g697b0d99.el8.x86 945 kB/s | 94 kB 00:00 2022-11-27T17:43:27.133 INFO:teuthology.orchestra.run.smithi099.stdout:(132/133): librados2-18.0.0-594.g697b0d99.el8.x 3.0 MB/s | 3.6 MB 00:01 2022-11-27T17:43:27.325 INFO:teuthology.orchestra.run.smithi119.stdout:(25/133): librgw2-18.0.0-594.g697b0d99.el8.x86_ 2.8 MB/s | 6.5 MB 00:02 2022-11-27T17:43:27.467 INFO:teuthology.orchestra.run.smithi099.stdout:(133/133): librbd1-18.0.0-594.g697b0d99.el8.x86 2.4 MB/s | 3.6 MB 00:01 2022-11-27T17:43:27.470 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2022-11-27T17:43:27.470 INFO:teuthology.orchestra.run.smithi099.stdout:Total 16 MB/s | 219 MB 00:13 2022-11-27T17:43:27.475 INFO:teuthology.orchestra.run.smithi119.stdout:(26/133): rbd-nbd-18.0.0-594.g697b0d99.el8.x86_ 1.1 MB/s | 175 kB 00:00 2022-11-27T17:43:27.583 INFO:teuthology.orchestra.run.smithi119.stdout:(27/133): rbd-mirror-18.0.0-594.g697b0d99.el8.x 2.7 MB/s | 3.5 MB 00:01 2022-11-27T17:43:27.609 INFO:teuthology.orchestra.run.smithi119.stdout:(28/133): ceph-grafana-dashboards-18.0.0-594.g6 171 kB/s | 23 kB 00:00 2022-11-27T17:43:27.659 INFO:teuthology.orchestra.run.smithi119.stdout:(29/133): ceph-mgr-cephadm-18.0.0-594.g697b0d99 1.5 MB/s | 118 kB 00:00 2022-11-27T17:43:28.285 INFO:teuthology.orchestra.run.smithi119.stdout:(30/133): ceph-mgr-dashboard-18.0.0-594.g697b0d 2.4 MB/s | 1.7 MB 00:00 2022-11-27T17:43:28.324 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2022-11-27T17:43:28.479 INFO:teuthology.orchestra.run.smithi119.stdout:(31/133): ceph-mgr-modules-core-18.0.0-594.g697 1.2 MB/s | 234 kB 00:00 2022-11-27T17:43:28.482 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2022-11-27T17:43:28.483 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2022-11-27T17:43:28.584 INFO:teuthology.orchestra.run.smithi119.stdout:(32/133): ceph-mgr-rook-18.0.0-594.g697b0d99.el 483 kB/s | 47 kB 00:00 2022-11-27T17:43:28.669 INFO:teuthology.orchestra.run.smithi119.stdout:(33/133): ceph-prometheus-alerts-18.0.0-594.g69 157 kB/s | 14 kB 00:00 2022-11-27T17:43:28.836 INFO:teuthology.orchestra.run.smithi119.stdout:(34/133): ceph-volume-18.0.0-594.g697b0d99.el8. 1.5 MB/s | 263 kB 00:00 2022-11-27T17:43:28.928 INFO:teuthology.orchestra.run.smithi119.stdout:(35/133): cephadm-18.0.0-594.g697b0d99.el8.noar 889 kB/s | 81 kB 00:00 2022-11-27T17:43:29.028 INFO:teuthology.orchestra.run.smithi119.stdout:(36/133): cephfs-top-18.0.0-594.g697b0d99.el8.n 154 kB/s | 15 kB 00:00 2022-11-27T17:43:29.220 INFO:teuthology.orchestra.run.smithi119.stdout:(37/133): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2022-11-27T17:43:29.270 INFO:teuthology.orchestra.run.smithi119.stdout:(38/133): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2022-11-27T17:43:29.412 INFO:teuthology.orchestra.run.smithi119.stdout:(39/133): libunwind-1.3.1-3.el8.x86_64.rpm 530 kB/s | 75 kB 00:00 2022-11-27T17:43:29.496 INFO:teuthology.orchestra.run.smithi119.stdout:(40/133): python3-asyncssh-2.7.0-2.el8.noarch.r 5.6 MB/s | 476 kB 00:00 2022-11-27T17:43:29.546 INFO:teuthology.orchestra.run.smithi119.stdout:(41/133): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 876 kB/s | 44 kB 00:00 2022-11-27T17:43:29.605 INFO:teuthology.orchestra.run.smithi119.stdout:(42/133): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2022-11-27T17:43:29.635 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2022-11-27T17:43:29.639 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2022-11-27T17:43:29.655 INFO:teuthology.orchestra.run.smithi119.stdout:(43/133): python3-cachetools-3.1.1-4.el8.noarch 659 kB/s | 33 kB 00:00 2022-11-27T17:43:29.705 INFO:teuthology.orchestra.run.smithi119.stdout:(44/133): python3-certifi-2018.10.15-7.el8.noar 328 kB/s | 16 kB 00:00 2022-11-27T17:43:29.764 INFO:teuthology.orchestra.run.smithi119.stdout:(45/133): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2022-11-27T17:43:29.923 INFO:teuthology.orchestra.run.smithi119.stdout:(46/133): python3-cherrypy-18.4.0-1.el8.noarch. 2.4 MB/s | 384 kB 00:00 2022-11-27T17:43:30.006 INFO:teuthology.orchestra.run.smithi119.stdout:(47/133): python3-google-auth-1.1.1-10.el8.noar 975 kB/s | 81 kB 00:00 2022-11-27T17:43:30.090 INFO:teuthology.orchestra.run.smithi119.stdout:(48/133): ceph-mgr-diskprediction-local-18.0.0- 3.0 MB/s | 7.4 MB 00:02 2022-11-27T17:43:30.107 INFO:teuthology.orchestra.run.smithi119.stdout:(49/133): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2022-11-27T17:43:30.182 INFO:teuthology.orchestra.run.smithi119.stdout:(50/133): python3-jaraco-functools-2.0-4.el8.no 192 kB/s | 18 kB 00:00 2022-11-27T17:43:30.257 INFO:teuthology.orchestra.run.smithi119.stdout:(51/133): python3-logutils-0.3.5-11.el8.noarch. 651 kB/s | 49 kB 00:00 2022-11-27T17:43:30.333 INFO:teuthology.orchestra.run.smithi119.stdout:(52/133): python3-more-itertools-7.2.0-3.el8.no 784 kB/s | 59 kB 00:00 2022-11-27T17:43:30.408 INFO:teuthology.orchestra.run.smithi119.stdout:(53/133): python3-natsort-7.1.1-2.el8.noarch.rp 803 kB/s | 60 kB 00:00 2022-11-27T17:43:30.492 INFO:teuthology.orchestra.run.smithi119.stdout:(54/133): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2022-11-27T17:43:30.597 INFO:teuthology.orchestra.run.smithi119.stdout:(55/133): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2022-11-27T17:43:30.676 INFO:teuthology.orchestra.run.smithi119.stdout:(56/133): python3-kubernetes-11.0.0-6.el8.noarc 2.1 MB/s | 1.2 MB 00:00 2022-11-27T17:43:30.693 INFO:teuthology.orchestra.run.smithi119.stdout:(57/133): python3-repoze-lru-0.7-6.el8.noarch.r 335 kB/s | 34 kB 00:00 2022-11-27T17:43:30.760 INFO:teuthology.orchestra.run.smithi119.stdout:(58/133): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2022-11-27T17:43:30.785 INFO:teuthology.orchestra.run.smithi119.stdout:(59/133): python3-rsa-4.8-1.el8.noarch.rpm 668 kB/s | 61 kB 00:00 2022-11-27T17:43:30.844 INFO:teuthology.orchestra.run.smithi119.stdout:(60/133): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2022-11-27T17:43:30.869 INFO:teuthology.orchestra.run.smithi119.stdout:(61/133): python3-singledispatch-3.4.0.3-18.el8 286 kB/s | 24 kB 00:00 2022-11-27T17:43:30.927 INFO:teuthology.orchestra.run.smithi119.stdout:(62/133): python3-tempora-1.14.1-5.el8.noarch.r 350 kB/s | 29 kB 00:00 2022-11-27T17:43:30.953 INFO:teuthology.orchestra.run.smithi119.stdout:(63/133): python3-trustme-0.6.0-4.el8.noarch.rp 329 kB/s | 27 kB 00:00 2022-11-27T17:43:31.011 INFO:teuthology.orchestra.run.smithi119.stdout:(64/133): python3-waitress-1.2.1-2.el8.1.noarch 2.8 MB/s | 241 kB 00:00 2022-11-27T17:43:31.045 INFO:teuthology.orchestra.run.smithi119.stdout:(65/133): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2022-11-27T17:43:31.096 INFO:teuthology.orchestra.run.smithi119.stdout:(66/133): python3-websocket-client-0.56.0-5.el8 732 kB/s | 61 kB 00:00 2022-11-27T17:43:31.129 INFO:teuthology.orchestra.run.smithi119.stdout:(67/133): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2022-11-27T17:43:31.180 INFO:teuthology.orchestra.run.smithi119.stdout:(68/133): python3-zc-lockfile-2.0-2.el8.noarch. 274 kB/s | 23 kB 00:00 2022-11-27T17:43:31.205 INFO:teuthology.orchestra.run.smithi119.stdout:(69/133): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2022-11-27T17:43:31.255 INFO:teuthology.orchestra.run.smithi119.stdout:(70/133): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2022-11-27T17:43:31.305 INFO:teuthology.orchestra.run.smithi119.stdout:(71/133): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2022-11-27T17:43:31.339 INFO:teuthology.orchestra.run.smithi119.stdout:(72/133): libsepol-devel-2.9-3.el8.x86_64.rpm 2.6 MB/s | 87 kB 00:00 2022-11-27T17:43:31.381 INFO:teuthology.orchestra.run.smithi119.stdout:(73/133): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2022-11-27T17:43:31.414 INFO:teuthology.orchestra.run.smithi119.stdout:(74/133): libselinux-devel-2.9-5.el8.x86_64.rpm 5.9 MB/s | 200 kB 00:00 2022-11-27T17:43:31.448 INFO:teuthology.orchestra.run.smithi119.stdout:(75/133): libverto-devel-0.3.0-5.el8.x86_64.rpm 536 kB/s | 18 kB 00:00 2022-11-27T17:43:31.482 INFO:teuthology.orchestra.run.smithi119.stdout:(76/133): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2022-11-27T17:43:31.523 INFO:teuthology.orchestra.run.smithi119.stdout:(77/133): thrift-0.13.0-2.el8.x86_64.rpm 4.4 MB/s | 1.7 MB 00:00 2022-11-27T17:43:31.565 INFO:teuthology.orchestra.run.smithi119.stdout:(78/133): python3-cryptography-3.2.1-5.el8.x86_ 6.5 MB/s | 559 kB 00:00 2022-11-27T17:43:31.591 INFO:teuthology.orchestra.run.smithi119.stdout:(79/133): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2022-11-27T17:43:31.624 INFO:teuthology.orchestra.run.smithi119.stdout:(80/133): python3-pycparser-2.14-14.el8.noarch. 1.8 MB/s | 109 kB 00:00 2022-11-27T17:43:31.666 INFO:teuthology.orchestra.run.smithi119.stdout:(81/133): krb5-devel-1.18.2-14.el8.x86_64.rpm 7.4 MB/s | 560 kB 00:00 2022-11-27T17:43:31.700 INFO:teuthology.orchestra.run.smithi119.stdout:(82/133): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.2 MB/s | 171 kB 00:00 2022-11-27T17:43:31.725 INFO:teuthology.orchestra.run.smithi119.stdout:(83/133): python3-jwt-1.6.1-2.el8.noarch.rpm 741 kB/s | 43 kB 00:00 2022-11-27T17:43:31.751 INFO:teuthology.orchestra.run.smithi119.stdout:(84/133): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2022-11-27T17:43:31.768 INFO:teuthology.orchestra.run.smithi119.stdout:(85/133): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2022-11-27T17:43:31.802 INFO:teuthology.orchestra.run.smithi119.stdout:(86/133): libaio-devel-0.3.112-1.el8.x86_64.rpm 563 kB/s | 19 kB 00:00 2022-11-27T17:43:31.827 INFO:teuthology.orchestra.run.smithi119.stdout:(87/133): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2022-11-27T17:43:31.852 INFO:teuthology.orchestra.run.smithi119.stdout:(88/133): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2022-11-27T17:43:31.877 INFO:teuthology.orchestra.run.smithi119.stdout:(89/133): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2022-11-27T17:43:31.902 INFO:teuthology.orchestra.run.smithi119.stdout:(90/133): python3-requests-oauthlib-1.0.0-1.el8 852 kB/s | 43 kB 00:00 2022-11-27T17:43:31.928 INFO:teuthology.orchestra.run.smithi119.stdout:(91/133): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2022-11-27T17:43:31.953 INFO:teuthology.orchestra.run.smithi119.stdout:(92/133): libcom_err-devel-1.45.6-4.el8.x86_64. 768 kB/s | 39 kB 00:00 2022-11-27T17:43:32.004 INFO:teuthology.orchestra.run.smithi119.stdout:(93/133): keyutils-libs-devel-1.5.10-9.el8.x86_ 964 kB/s | 48 kB 00:00 2022-11-27T17:43:32.054 INFO:teuthology.orchestra.run.smithi119.stdout:(94/133): openssl-devel-1.1.1k-7.el8_6.x86_64.r 19 MB/s | 2.3 MB 00:00 2022-11-27T17:43:32.096 INFO:teuthology.orchestra.run.smithi119.stdout:(95/133): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.9 MB/s | 644 kB 00:00 2022-11-27T17:43:32.121 INFO:teuthology.orchestra.run.smithi119.stdout:(96/133): NetworkManager-initscripts-updown-1.3 2.0 MB/s | 138 kB 00:00 2022-11-27T17:43:32.163 INFO:teuthology.orchestra.run.smithi119.stdout:(97/133): postgresql-contrib-10.21-2.module+el8 12 MB/s | 808 kB 00:00 2022-11-27T17:43:32.197 INFO:teuthology.orchestra.run.smithi119.stdout:(98/133): bison-3.0.4-10.el8.x86_64.rpm 9.1 MB/s | 688 kB 00:00 2022-11-27T17:43:32.256 INFO:teuthology.orchestra.run.smithi119.stdout:(99/133): postgresql-10.21-2.module+el8.6.0+153 16 MB/s | 1.5 MB 00:00 2022-11-27T17:43:32.347 INFO:teuthology.orchestra.run.smithi119.stdout:(100/133): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2022-11-27T17:43:32.389 INFO:teuthology.orchestra.run.smithi119.stdout:(101/133): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2022-11-27T17:43:32.465 INFO:teuthology.orchestra.run.smithi119.stdout:(102/133): postgresql-server-10.21-2.module+el8 24 MB/s | 5.1 MB 00:00 2022-11-27T17:43:32.490 INFO:teuthology.orchestra.run.smithi119.stdout:(103/133): python3-markupsafe-0.23-19.el8.x86_6 388 kB/s | 39 kB 00:00 2022-11-27T17:43:32.532 INFO:teuthology.orchestra.run.smithi119.stdout:(104/133): python3-prettytable-0.7.2-14.el8.noa 658 kB/s | 44 kB 00:00 2022-11-27T17:43:32.565 INFO:teuthology.orchestra.run.smithi119.stdout:(105/133): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2022-11-27T17:43:32.599 INFO:teuthology.orchestra.run.smithi119.stdout:(106/133): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2022-11-27T17:43:32.633 INFO:teuthology.orchestra.run.smithi119.stdout:(107/133): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2022-11-27T17:43:32.666 INFO:teuthology.orchestra.run.smithi119.stdout:(108/133): python3-numpy-f2py-1.14.3-10.el8.x86 3.3 MB/s | 225 kB 00:00 2022-11-27T17:43:32.717 INFO:teuthology.orchestra.run.smithi119.stdout:(109/133): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-11-27T17:43:32.800 INFO:teuthology.orchestra.run.smithi119.stdout:(110/133): openblas-threads-0.3.15-3.el8.x86_64 29 MB/s | 4.9 MB 00:00 2022-11-27T17:43:32.842 INFO:teuthology.orchestra.run.smithi119.stdout:(111/133): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2022-11-27T17:43:32.918 INFO:teuthology.orchestra.run.smithi119.stdout:(112/133): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2022-11-27T17:43:32.943 INFO:teuthology.orchestra.run.smithi119.stdout:(113/133): socat-1.7.4.1-1.el8.x86_64.rpm 3.1 MB/s | 323 kB 00:00 2022-11-27T17:43:32.977 INFO:teuthology.orchestra.run.smithi119.stdout:(114/133): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2022-11-27T17:43:33.010 INFO:teuthology.orchestra.run.smithi119.stdout:(115/133): jq-1.6-3.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2022-11-27T17:43:33.085 INFO:teuthology.orchestra.run.smithi119.stdout:(116/133): openblas-0.3.15-3.el8.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2022-11-27T17:43:33.127 INFO:teuthology.orchestra.run.smithi119.stdout:(117/133): automake-1.16.1-7.el8.noarch.rpm 17 MB/s | 713 kB 00:00 2022-11-27T17:43:33.161 INFO:teuthology.orchestra.run.smithi119.stdout:(118/133): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2022-11-27T17:43:33.203 INFO:teuthology.orchestra.run.smithi119.stdout:(119/133): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.0 MB/s | 126 kB 00:00 2022-11-27T17:43:33.228 INFO:teuthology.orchestra.run.smithi119.stdout:(120/133): python3-jsonpointer-1.10-11.el8.noar 799 kB/s | 20 kB 00:00 2022-11-27T17:43:33.399 INFO:teuthology.orchestra.run.smithi119.stdout:(121/133): python3-scipy-1.0.0-21.module+el8.5. 35 MB/s | 14 MB 00:00 2022-11-27T17:43:33.429 INFO:teuthology.orchestra.run.smithi119.stdout:(122/133): libpq-13.5-1.el8.x86_64.rpm 988 kB/s | 198 kB 00:00 2022-11-27T17:43:33.463 INFO:teuthology.orchestra.run.smithi119.stdout:(123/133): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.0 MB/s | 259 kB 00:00 2022-11-27T17:43:33.488 INFO:teuthology.orchestra.run.smithi119.stdout:(124/133): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2022-11-27T17:43:33.521 INFO:teuthology.orchestra.run.smithi119.stdout:(125/133): python3-jsonpatch-1.21-2.el8.noarch. 454 kB/s | 27 kB 00:00 2022-11-27T17:43:33.547 INFO:teuthology.orchestra.run.smithi119.stdout:(126/133): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2022-11-27T17:43:33.572 INFO:teuthology.orchestra.run.smithi119.stdout:(127/133): python3-pyasn1-modules-0.3.7-6.el8.n 2.1 MB/s | 110 kB 00:00 2022-11-27T17:43:35.014 INFO:teuthology.orchestra.run.smithi119.stdout:(128/133): librbd1-18.0.0-594.g697b0d99.el8.x86 2.5 MB/s | 3.6 MB 00:01 2022-11-27T17:43:35.064 INFO:teuthology.orchestra.run.smithi119.stdout:(129/133): openssl-1.1.1k-7.el8_6.x86_64.rpm 14 MB/s | 709 kB 00:00 2022-11-27T17:43:35.123 INFO:teuthology.orchestra.run.smithi119.stdout:(130/133): openssl-libs-1.1.1k-7.el8_6.x86_64.r 25 MB/s | 1.5 MB 00:00 2022-11-27T17:43:35.174 INFO:teuthology.orchestra.run.smithi119.stdout:(131/133): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2022-11-27T17:43:35.174 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2022-11-27T17:43:35.372 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/138 2022-11-27T17:43:35.588 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/138 2022-11-27T17:43:35.764 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/138 2022-11-27T17:43:35.964 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/138 2022-11-27T17:43:36.125 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/138 2022-11-27T17:43:36.193 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/138 2022-11-27T17:43:36.248 INFO:teuthology.orchestra.run.smithi119.stdout:(132/133): librados2-18.0.0-594.g697b0d99.el8.x 1.3 MB/s | 3.6 MB 00:02 2022-11-27T17:43:36.408 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/138 2022-11-27T17:43:36.566 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/138 2022-11-27T17:43:36.982 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/138 2022-11-27T17:43:37.233 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:43:37.605 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:43:37.833 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:43:37.959 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:43:38.173 INFO:teuthology.orchestra.run.smithi119.stdout:(133/133): ceph-test-18.0.0-594.g697b0d99.el8.x 3.8 MB/s | 55 MB 00:14 2022-11-27T17:43:38.176 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2022-11-27T17:43:38.177 INFO:teuthology.orchestra.run.smithi119.stdout:Total 9.5 MB/s | 219 MB 00:23 2022-11-27T17:43:38.241 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:43:38.340 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:43:38.572 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 10/138 2022-11-27T17:43:38.829 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libpq-13.5-1.el8.x86_64 11/138 2022-11-27T17:43:39.027 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/138 2022-11-27T17:43:39.041 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2022-11-27T17:43:39.202 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2022-11-27T17:43:39.202 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2022-11-27T17:43:39.392 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/138 2022-11-27T17:43:39.645 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/138 2022-11-27T17:43:39.807 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:43:39.912 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:43:40.115 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 16/138 2022-11-27T17:43:40.185 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 16/138 2022-11-27T17:43:40.337 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 17/138 2022-11-27T17:43:40.412 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2022-11-27T17:43:40.416 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2022-11-27T17:43:40.490 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 18/138 2022-11-27T17:43:40.632 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 18/138 2022-11-27T17:43:40.782 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 19/138 2022-11-27T17:43:41.400 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 19/138 2022-11-27T17:43:41.590 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 20/138 2022-11-27T17:43:42.188 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 20/138 2022-11-27T17:43:42.390 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.15-3.el8.x86_64 21/138 2022-11-27T17:43:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 21/138 2022-11-27T17:43:43.540 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/138 2022-11-27T17:43:43.703 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 23/138 2022-11-27T17:43:44.011 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/138 2022-11-27T17:43:44.322 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 25/138 2022-11-27T17:43:44.571 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/138 2022-11-27T17:43:44.721 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x 27/138 2022-11-27T17:43:44.842 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8 28/138 2022-11-27T17:43:45.072 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 29/138 2022-11-27T17:43:45.335 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2022-11-27T17:43:45.598 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/138 2022-11-27T17:43:45.874 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/138 2022-11-27T17:43:46.006 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/138 2022-11-27T17:43:46.086 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/138 2022-11-27T17:43:46.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/138 2022-11-27T17:43:46.351 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/138 2022-11-27T17:43:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 4/138 2022-11-27T17:43:46.555 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 5/138 2022-11-27T17:43:46.600 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/138 2022-11-27T17:43:46.656 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 5/138 2022-11-27T17:43:46.814 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/138 2022-11-27T17:43:46.838 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/138 2022-11-27T17:43:47.217 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/138 2022-11-27T17:43:47.496 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:43:47.890 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:43:48.071 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:43:48.202 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:43:48.388 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:43:48.482 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:43:48.528 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/138 2022-11-27T17:43:48.660 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 10/138 2022-11-27T17:43:48.898 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpq-13.5-1.el8.x86_64 11/138 2022-11-27T17:43:48.906 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 34/138 2022-11-27T17:43:49.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/138 2022-11-27T17:43:49.150 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/138 2022-11-27T17:43:49.331 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/138 2022-11-27T17:43:49.432 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/138 2022-11-27T17:43:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 37/138 2022-11-27T17:43:49.658 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/138 2022-11-27T17:43:49.800 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 38/138 2022-11-27T17:43:49.813 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:43:49.929 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:43:50.000 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/138 2022-11-27T17:43:50.071 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 16/138 2022-11-27T17:43:50.142 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 16/138 2022-11-27T17:43:50.159 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 40/138 2022-11-27T17:43:50.316 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 41/138 2022-11-27T17:43:50.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 17/138 2022-11-27T17:43:50.537 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 18/138 2022-11-27T17:43:50.537 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/138 2022-11-27T17:43:50.735 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 43/138 2022-11-27T17:43:50.757 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/138 2022-11-27T17:43:50.918 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/138 2022-11-27T17:43:51.002 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/138 2022-11-27T17:43:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 21/138 2022-11-27T17:43:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 21/138 2022-11-27T17:43:51.312 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 22/138 2022-11-27T17:43:51.417 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 45/138 2022-11-27T17:43:51.757 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 46/138 2022-11-27T17:43:51.914 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 22/138 2022-11-27T17:43:51.998 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 47/138 2022-11-27T17:43:52.112 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 23/138 2022-11-27T17:43:52.217 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 48/138 2022-11-27T17:43:52.475 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/138 2022-11-27T17:43:52.714 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 23/138 2022-11-27T17:43:52.786 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/138 2022-11-27T17:43:52.871 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.15-3.el8.x86_64 24/138 2022-11-27T17:43:52.979 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/138 2022-11-27T17:43:53.197 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/138 2022-11-27T17:43:53.415 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/138 2022-11-27T17:43:53.544 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 24/138 2022-11-27T17:43:53.574 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 54/138 2022-11-27T17:43:53.875 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/138 2022-11-27T17:43:53.930 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/138 2022-11-27T17:43:54.125 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/138 2022-11-27T17:43:54.149 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/138 2022-11-27T17:43:54.326 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x 27/138 2022-11-27T17:43:54.373 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 57/138 2022-11-27T17:43:54.472 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8 28/138 2022-11-27T17:43:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/138 2022-11-27T17:43:54.748 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 29/138 2022-11-27T17:43:54.872 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 59/138 2022-11-27T17:43:54.954 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/138 2022-11-27T17:43:55.071 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-15.el8.x86_64 60/138 2022-11-27T17:43:55.262 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/138 2022-11-27T17:43:55.372 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/138 2022-11-27T17:43:55.547 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/138 2022-11-27T17:43:55.625 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/138 2022-11-27T17:43:55.845 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-3.el8.x86_64 63/138 2022-11-27T17:43:56.012 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/138 2022-11-27T17:43:56.293 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/138 2022-11-27T17:43:56.537 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : automake-1.16.1-7.el8.noarch 66/138 2022-11-27T17:43:56.697 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/138 2022-11-27T17:43:56.697 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 67/138 2022-11-27T17:43:56.912 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/138 2022-11-27T17:43:57.071 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 31/138 2022-11-27T17:43:57.074 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/138 2022-11-27T17:43:57.379 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/138 2022-11-27T17:43:57.569 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/138 2022-11-27T17:43:57.791 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/138 2022-11-27T17:43:57.939 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/138 2022-11-27T17:43:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/138 2022-11-27T17:43:58.171 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/138 2022-11-27T17:43:58.349 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/138 2022-11-27T17:43:58.406 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/138 2022-11-27T17:43:58.548 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/138 2022-11-27T17:43:58.695 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/138 2022-11-27T17:43:58.706 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/138 2022-11-27T17:43:58.892 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/138 2022-11-27T17:43:58.947 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/138 2022-11-27T17:43:59.137 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/138 2022-11-27T17:43:59.172 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/138 2022-11-27T17:43:59.341 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 37/138 2022-11-27T17:43:59.364 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/138 2022-11-27T17:43:59.559 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/138 2022-11-27T17:43:59.573 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 38/138 2022-11-27T17:43:59.778 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/138 2022-11-27T17:43:59.788 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/138 2022-11-27T17:43:59.961 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/138 2022-11-27T17:43:59.993 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/138 2022-11-27T17:44:00.041 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/138 2022-11-27T17:44:00.151 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/138 2022-11-27T17:44:00.236 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 41/138 2022-11-27T17:44:00.315 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 41/138 2022-11-27T17:44:00.376 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/138 2022-11-27T17:44:00.492 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-3.el8.x86_64 42/138 2022-11-27T17:44:00.651 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 43/138 2022-11-27T17:44:00.889 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 44/138 2022-11-27T17:44:01.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.1-7.el8.noarch 45/138 2022-11-27T17:44:01.256 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/138 2022-11-27T17:44:01.406 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 47/138 2022-11-27T17:44:01.409 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/138 2022-11-27T17:44:01.614 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 48/138 2022-11-27T17:44:01.672 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/138 2022-11-27T17:44:01.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 49/138 2022-11-27T17:44:01.915 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/138 2022-11-27T17:44:02.015 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 50/138 2022-11-27T17:44:02.217 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/138 2022-11-27T17:44:02.263 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/138 2022-11-27T17:44:02.561 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/138 2022-11-27T17:44:02.666 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/138 2022-11-27T17:44:02.735 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/138 2022-11-27T17:44:03.001 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/138 2022-11-27T17:44:03.163 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/138 2022-11-27T17:44:03.405 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/138 2022-11-27T17:44:03.532 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/138 2022-11-27T17:44:03.656 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/138 2022-11-27T17:44:03.718 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 93/138 2022-11-27T17:44:03.796 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 93/138 2022-11-27T17:44:04.000 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/138 2022-11-27T17:44:04.044 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 94/138 2022-11-27T17:44:04.243 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/138 2022-11-27T17:44:04.436 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/138 2022-11-27T17:44:04.663 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/138 2022-11-27T17:44:04.796 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 61/138 2022-11-27T17:44:05.125 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 62/138 2022-11-27T17:44:05.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/138 2022-11-27T17:44:05.589 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 64/138 2022-11-27T17:44:05.847 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/138 2022-11-27T17:44:06.077 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/138 2022-11-27T17:44:06.586 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.7-15.el8.x86_64 67/138 2022-11-27T17:44:06.760 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/138 2022-11-27T17:44:06.939 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/138 2022-11-27T17:44:07.219 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/138 2022-11-27T17:44:07.467 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/138 2022-11-27T17:44:07.648 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/138 2022-11-27T17:44:07.763 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:07.804 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/138 2022-11-27T17:44:07.985 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:08.028 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/138 2022-11-27T17:44:08.239 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/138 2022-11-27T17:44:08.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/138 2022-11-27T17:44:08.541 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:08.596 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/138 2022-11-27T17:44:08.765 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/138 2022-11-27T17:44:08.832 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 96/138 2022-11-27T17:44:08.903 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 96/138 2022-11-27T17:44:08.980 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/138 2022-11-27T17:44:09.017 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 97/138 2022-11-27T17:44:09.179 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/138 2022-11-27T17:44:09.382 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/138 2022-11-27T17:44:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/138 2022-11-27T17:44:09.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/138 2022-11-27T17:44:09.992 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/138 2022-11-27T17:44:10.226 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/138 2022-11-27T17:44:11.272 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/138 2022-11-27T17:44:11.554 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/138 2022-11-27T17:44:11.775 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/138 2022-11-27T17:44:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/138 2022-11-27T17:44:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/138 2022-11-27T17:44:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/138 2022-11-27T17:44:13.351 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/138 2022-11-27T17:44:13.500 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 93/138 2022-11-27T17:44:13.574 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 93/138 2022-11-27T17:44:13.776 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 94/138 2022-11-27T17:44:17.583 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:18.405 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 95/138 2022-11-27T17:44:18.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 96/138 2022-11-27T17:44:18.768 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 96/138 2022-11-27T17:44:18.900 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 97/138 2022-11-27T17:44:27.645 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 97/138 2022-11-27T17:44:27.645 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2022-11-27T17:44:27.646 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2022-11-27T17:44:27.646 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2022-11-27T17:44:27.646 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2022-11-27T17:44:27.646 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2022-11-27T17:44:27.647 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2022-11-27T17:44:27.647 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2022-11-27T17:44:27.647 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2022-11-27T17:44:27.647 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2022-11-27T17:44:27.648 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:44:28.022 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 98/138 2022-11-27T17:44:29.502 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 98/138 2022-11-27T17:44:29.710 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 99/138 2022-11-27T17:44:29.927 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 99/138 2022-11-27T17:44:30.113 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 100/138 2022-11-27T17:44:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 100/138 2022-11-27T17:44:30.663 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 101/138 2022-11-27T17:44:30.680 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 101/138 2022-11-27T17:44:32.284 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 102/138 2022-11-27T17:44:32.454 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 102/138 2022-11-27T17:44:32.568 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el 103/138 2022-11-27T17:44:32.723 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.e 104/138 2022-11-27T17:44:32.887 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 105/138 2022-11-27T17:44:33.097 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 105/138 2022-11-27T17:44:33.343 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 106/138 2022-11-27T17:44:34.300 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 106/138 2022-11-27T17:44:34.462 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 107/138 2022-11-27T17:44:34.550 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 107/138 2022-11-27T17:44:34.936 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8 108/138 2022-11-27T17:44:35.111 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 109/138 2022-11-27T17:44:35.145 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 109/138 2022-11-27T17:44:35.345 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 110/138 2022-11-27T17:44:35.356 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 110/138 2022-11-27T17:44:37.072 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 111/138 2022-11-27T17:44:37.288 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 112/138 2022-11-27T17:44:37.321 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 112/138 2022-11-27T17:44:37.453 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-594.g697b0d 113/138 2022-11-27T17:44:37.494 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-594.g697b0d 113/138 2022-11-27T17:44:37.536 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 97/138 2022-11-27T17:44:37.536 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2022-11-27T17:44:37.537 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2022-11-27T17:44:37.537 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2022-11-27T17:44:37.537 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /run 2022-11-27T17:44:37.537 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2022-11-27T17:44:37.537 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2022-11-27T17:44:37.538 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2022-11-27T17:44:37.538 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2022-11-27T17:44:37.538 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2022-11-27T17:44:37.538 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:44:37.669 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 114/138 2022-11-27T17:44:37.939 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 98/138 2022-11-27T17:44:38.015 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 114/138 2022-11-27T17:44:38.144 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 115/138 2022-11-27T17:44:39.438 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 98/138 2022-11-27T17:44:39.631 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 99/138 2022-11-27T17:44:39.851 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 99/138 2022-11-27T17:44:40.038 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 100/138 2022-11-27T17:44:40.423 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 100/138 2022-11-27T17:44:40.588 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 101/138 2022-11-27T17:44:40.606 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 101/138 2022-11-27T17:44:42.244 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 102/138 2022-11-27T17:44:42.417 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 102/138 2022-11-27T17:44:42.560 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el 103/138 2022-11-27T17:44:42.760 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.e 104/138 2022-11-27T17:44:42.962 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 105/138 2022-11-27T17:44:43.166 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 105/138 2022-11-27T17:44:43.226 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 115/138 2022-11-27T17:44:43.372 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 106/138 2022-11-27T17:44:43.512 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 116/138 2022-11-27T17:44:43.685 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libtool-2.4.6-25.el8.x86_64 117/138 2022-11-27T17:44:43.874 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 117/138 2022-11-27T17:44:44.174 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 118/138 2022-11-27T17:44:44.339 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 106/138 2022-11-27T17:44:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 107/138 2022-11-27T17:44:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 107/138 2022-11-27T17:44:44.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8 108/138 2022-11-27T17:44:45.007 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 119/138 2022-11-27T17:44:45.110 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 109/138 2022-11-27T17:44:45.145 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 109/138 2022-11-27T17:44:45.279 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 120/138 2022-11-27T17:44:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 110/138 2022-11-27T17:44:45.323 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 110/138 2022-11-27T17:44:45.464 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 121/138 2022-11-27T17:44:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_6 122/138 2022-11-27T17:44:45.802 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/138 2022-11-27T17:44:47.082 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 111/138 2022-11-27T17:44:47.289 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 112/138 2022-11-27T17:44:47.324 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 112/138 2022-11-27T17:44:47.479 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-594.g697b0d 113/138 2022-11-27T17:44:47.521 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-594.g697b0d 113/138 2022-11-27T17:44:47.670 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 114/138 2022-11-27T17:44:47.903 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:48.020 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 114/138 2022-11-27T17:44:48.170 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 115/138 2022-11-27T17:44:48.209 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:48.239 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:48.501 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 125/138 2022-11-27T17:44:48.683 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 126/138 2022-11-27T17:44:48.975 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 127/138 2022-11-27T17:44:49.356 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/138 2022-11-27T17:44:49.702 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 129/138 2022-11-27T17:44:49.858 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/138 2022-11-27T17:44:49.866 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/138 2022-11-27T17:44:50.044 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/138 2022-11-27T17:44:50.308 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flex-2.6.1-9.el8.x86_64 132/138 2022-11-27T17:44:50.382 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 132/138 2022-11-27T17:44:50.583 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : bison-3.0.4-10.el8.x86_64 133/138 2022-11-27T17:44:50.597 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 133/138 2022-11-27T17:44:50.810 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 134/138 2022-11-27T17:44:50.810 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 135/138 2022-11-27T17:44:51.003 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 135/138 2022-11-27T17:44:51.004 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 136/138 2022-11-27T17:44:51.277 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 136/138 2022-11-27T17:44:51.278 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 137/138 2022-11-27T17:44:51.557 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 137/138 2022-11-27T17:44:51.723 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:44:52.311 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:44:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 1/138 2022-11-27T17:44:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 2/138 2022-11-27T17:44:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 3/138 2022-11-27T17:44:52.312 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 4/138 2022-11-27T17:44:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-594.g697b0d 5/138 2022-11-27T17:44:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 6/138 2022-11-27T17:44:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:44:52.313 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:44:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:44:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 10/138 2022-11-27T17:44:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 11/138 2022-11-27T17:44:52.314 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 12/138 2022-11-27T17:44:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 13/138 2022-11-27T17:44:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_6 14/138 2022-11-27T17:44:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:44:52.315 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 16/138 2022-11-27T17:44:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 17/138 2022-11-27T17:44:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 18/138 2022-11-27T17:44:52.316 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 19/138 2022-11-27T17:44:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8 20/138 2022-11-27T17:44:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x 21/138 2022-11-27T17:44:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 22/138 2022-11-27T17:44:52.317 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 23/138 2022-11-27T17:44:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 24/138 2022-11-27T17:44:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 25/138 2022-11-27T17:44:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 26/138 2022-11-27T17:44:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 27/138 2022-11-27T17:44:52.318 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 28/138 2022-11-27T17:44:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.e 29/138 2022-11-27T17:44:52.319 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 30/138 2022-11-27T17:44:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 31/138 2022-11-27T17:44:52.320 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 32/138 2022-11-27T17:44:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8 33/138 2022-11-27T17:44:52.321 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 34/138 2022-11-27T17:44:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el 35/138 2022-11-27T17:44:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 36/138 2022-11-27T17:44:52.322 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 37/138 2022-11-27T17:44:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 38/138 2022-11-27T17:44:52.323 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/138 2022-11-27T17:44:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/138 2022-11-27T17:44:52.324 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/138 2022-11-27T17:44:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/138 2022-11-27T17:44:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/138 2022-11-27T17:44:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/138 2022-11-27T17:44:52.326 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/138 2022-11-27T17:44:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/138 2022-11-27T17:44:52.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/138 2022-11-27T17:44:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/138 2022-11-27T17:44:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/138 2022-11-27T17:44:52.328 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/138 2022-11-27T17:44:52.329 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/138 2022-11-27T17:44:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/138 2022-11-27T17:44:52.330 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/138 2022-11-27T17:44:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/138 2022-11-27T17:44:52.331 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/138 2022-11-27T17:44:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/138 2022-11-27T17:44:52.332 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/138 2022-11-27T17:44:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/138 2022-11-27T17:44:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/138 2022-11-27T17:44:52.333 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/138 2022-11-27T17:44:52.334 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/138 2022-11-27T17:44:52.334 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/138 2022-11-27T17:44:52.335 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/138 2022-11-27T17:44:52.335 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/138 2022-11-27T17:44:52.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/138 2022-11-27T17:44:52.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/138 2022-11-27T17:44:52.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/138 2022-11-27T17:44:52.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/138 2022-11-27T17:44:52.338 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/138 2022-11-27T17:44:52.338 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/138 2022-11-27T17:44:52.339 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/138 2022-11-27T17:44:52.339 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 72/138 2022-11-27T17:44:52.339 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : bison-3.0.4-10.el8.x86_64 73/138 2022-11-27T17:44:52.340 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 74/138 2022-11-27T17:44:52.340 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/138 2022-11-27T17:44:52.341 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 76/138 2022-11-27T17:44:52.341 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/138 2022-11-27T17:44:52.342 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 78/138 2022-11-27T17:44:52.342 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/138 2022-11-27T17:44:52.343 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 80/138 2022-11-27T17:44:52.343 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 81/138 2022-11-27T17:44:52.344 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 82/138 2022-11-27T17:44:52.344 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 83/138 2022-11-27T17:44:52.345 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 84/138 2022-11-27T17:44:52.345 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/138 2022-11-27T17:44:52.345 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/138 2022-11-27T17:44:52.346 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/138 2022-11-27T17:44:52.346 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/138 2022-11-27T17:44:52.347 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flex-2.6.1-9.el8.x86_64 89/138 2022-11-27T17:44:52.347 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 90/138 2022-11-27T17:44:52.348 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-3.el8.x86_64 91/138 2022-11-27T17:44:52.348 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 92/138 2022-11-27T17:44:52.349 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : automake-1.16.1-7.el8.noarch 93/138 2022-11-27T17:44:52.349 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 94/138 2022-11-27T17:44:52.350 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 95/138 2022-11-27T17:44:52.350 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 96/138 2022-11-27T17:44:52.350 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libpq-13.5-1.el8.x86_64 97/138 2022-11-27T17:44:52.351 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 98/138 2022-11-27T17:44:52.351 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 99/138 2022-11-27T17:44:52.352 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/138 2022-11-27T17:44:52.352 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 101/138 2022-11-27T17:44:52.353 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/138 2022-11-27T17:44:52.353 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 103/138 2022-11-27T17:44:52.354 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 104/138 2022-11-27T17:44:52.354 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/138 2022-11-27T17:44:52.355 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/138 2022-11-27T17:44:52.355 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 107/138 2022-11-27T17:44:52.355 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/138 2022-11-27T17:44:52.356 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 109/138 2022-11-27T17:44:52.356 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 110/138 2022-11-27T17:44:52.357 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/138 2022-11-27T17:44:52.357 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 112/138 2022-11-27T17:44:52.358 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 113/138 2022-11-27T17:44:52.358 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 114/138 2022-11-27T17:44:52.359 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 115/138 2022-11-27T17:44:52.359 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/138 2022-11-27T17:44:52.359 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 117/138 2022-11-27T17:44:52.360 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/138 2022-11-27T17:44:52.360 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 119/138 2022-11-27T17:44:52.361 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 120/138 2022-11-27T17:44:52.361 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 121/138 2022-11-27T17:44:52.362 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 122/138 2022-11-27T17:44:52.362 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 123/138 2022-11-27T17:44:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 124/138 2022-11-27T17:44:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 125/138 2022-11-27T17:44:52.363 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 126/138 2022-11-27T17:44:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 127/138 2022-11-27T17:44:52.364 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/138 2022-11-27T17:44:52.365 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 129/138 2022-11-27T17:44:52.365 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/138 2022-11-27T17:44:52.366 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 131/138 2022-11-27T17:44:52.366 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/138 2022-11-27T17:44:52.367 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/138 2022-11-27T17:44:52.367 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/138 2022-11-27T17:44:52.367 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/138 2022-11-27T17:44:52.368 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/138 2022-11-27T17:44:52.368 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 137/138 2022-11-27T17:44:53.308 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 115/138 2022-11-27T17:44:53.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 116/138 2022-11-27T17:44:54.323 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:44:54.323 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2022-11-27T17:44:54.407 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 117/138 2022-11-27T17:44:54.575 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:44:54.576 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2022-11-27T17:44:54.576 INFO:teuthology.orchestra.run.smithi099.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2022-11-27T17:44:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.577 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:44:54.578 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:44:54.578 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2022-11-27T17:44:54.578 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2022-11-27T17:44:54.578 INFO:teuthology.orchestra.run.smithi099.stdout: automake-1.16.1-7.el8.noarch 2022-11-27T17:44:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: bison-3.0.4-10.el8.x86_64 2022-11-27T17:44:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.579 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.580 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.581 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.583 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.584 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.585 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.586 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.587 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.588 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:44:54.589 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-11-27T17:44:54.589 INFO:teuthology.orchestra.run.smithi099.stdout: flex-2.6.1-9.el8.x86_64 2022-11-27T17:44:54.589 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-2.9.7-15.el8.x86_64 2022-11-27T17:44:54.589 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-11-27T17:44:54.590 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-3.el8.x86_64 2022-11-27T17:44:54.590 INFO:teuthology.orchestra.run.smithi099.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-11-27T17:44:54.590 INFO:teuthology.orchestra.run.smithi099.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-11-27T17:44:54.590 INFO:teuthology.orchestra.run.smithi099.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-11-27T17:44:54.591 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.591 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.591 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.591 INFO:teuthology.orchestra.run.smithi099.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-11-27T17:44:54.592 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-11-27T17:44:54.592 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2022-11-27T17:44:54.592 INFO:teuthology.orchestra.run.smithi099.stdout: libpq-13.5-1.el8.x86_64 2022-11-27T17:44:54.592 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-11-27T17:44:54.592 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-11-27T17:44:54.593 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.593 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.593 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-11-27T17:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-11-27T17:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-11-27T17:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libtool-2.4.6-25.el8.x86_64 2022-11-27T17:44:54.594 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2022-11-27T17:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-11-27T17:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-11-27T17:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-11-27T17:44:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-11-27T17:44:54.596 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.15-3.el8.x86_64 2022-11-27T17:44:54.596 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-11-27T17:44:54.596 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:44:54.596 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-11-27T17:44:54.597 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-11-27T17:44:54.597 INFO:teuthology.orchestra.run.smithi099.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-11-27T17:44:54.597 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:44:54.597 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:44:54.597 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:44:54.598 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-11-27T17:44:54.598 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.5.1-7.el8.noarch 2022-11-27T17:44:54.598 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-11-27T17:44:54.598 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-11-27T17:44:54.599 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-11-27T17:44:54.599 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.599 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.600 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.600 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-11-27T17:44:54.601 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-11-27T17:44:54.601 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-11-27T17:44:54.601 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-11-27T17:44:54.602 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-11-27T17:44:54.602 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-11-27T17:44:54.602 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2022-11-27T17:44:54.602 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-11-27T17:44:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-11-27T17:44:54.603 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-11-27T17:44:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-11-27T17:44:54.604 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-11-27T17:44:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-11-27T17:44:54.605 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-11-27T17:44:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-13.el8.noarch 2022-11-27T17:44:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-11-27T17:44:54.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-11-27T17:44:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-11-27T17:44:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-11-27T17:44:54.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-11-27T17:44:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-11-27T17:44:54.608 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-11-27T17:44:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2022-11-27T17:44:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-11-27T17:44:54.609 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-11-27T17:44:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-11-27T17:44:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-11-27T17:44:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2022-11-27T17:44:54.610 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-9.el8.noarch 2022-11-27T17:44:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-11-27T17:44:54.611 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-11-27T17:44:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.4.1-12.el8.noarch 2022-11-27T17:44:54.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.8-1.el8.noarch 2022-11-27T17:44:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-11-27T17:44:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-11-27T17:44:54.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-11-27T17:44:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-11-27T17:44:54.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-11-27T17:44:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-11-27T17:44:54.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-11-27T17:44:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-11-27T17:44:54.616 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-11-27T17:44:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-11-27T17:44:54.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-11-27T17:44:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:44:54.618 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-1.el8.x86_64 2022-11-27T17:44:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.13.0-2.el8.x86_64 2022-11-27T17:44:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: uuid-1.6.2-43.el8.x86_64 2022-11-27T17:44:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-11-27T17:44:54.619 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-11-27T17:44:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-11-27T17:44:54.620 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:44:54.620 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2022-11-27T17:44:54.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/138 2022-11-27T17:44:54.761 DEBUG:teuthology.parallel:result is None 2022-11-27T17:44:54.879 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 119/138 2022-11-27T17:44:55.019 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/138 2022-11-27T17:44:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/138 2022-11-27T17:44:55.440 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 121/138 2022-11-27T17:44:55.664 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_6 122/138 2022-11-27T17:44:55.811 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/138 2022-11-27T17:44:57.860 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:58.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:58.120 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/138 2022-11-27T17:44:58.283 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 125/138 2022-11-27T17:44:58.534 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 126/138 2022-11-27T17:44:58.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 127/138 2022-11-27T17:44:59.150 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/138 2022-11-27T17:44:59.487 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 129/138 2022-11-27T17:44:59.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/138 2022-11-27T17:44:59.759 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/138 2022-11-27T17:44:59.993 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/138 2022-11-27T17:45:00.005 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/138 2022-11-27T17:45:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 132/138 2022-11-27T17:45:00.176 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 132/138 2022-11-27T17:45:00.321 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 133/138 2022-11-27T17:45:00.512 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 134/138 2022-11-27T17:45:00.512 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 135/138 2022-11-27T17:45:00.712 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 135/138 2022-11-27T17:45:00.712 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 136/138 2022-11-27T17:45:00.953 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 136/138 2022-11-27T17:45:00.953 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 137/138 2022-11-27T17:45:01.233 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 137/138 2022-11-27T17:45:01.417 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:45:01.956 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:45:01.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:18.0.0-594.g697b0d99.el8.x86_64 1/138 2022-11-27T17:45:01.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 2/138 2022-11-27T17:45:01.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 3/138 2022-11-27T17:45:01.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 4/138 2022-11-27T17:45:01.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-594.g697b0d 5/138 2022-11-27T17:45:01.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 6/138 2022-11-27T17:45:01.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 7/138 2022-11-27T17:45:01.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 8/138 2022-11-27T17:45:01.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 9/138 2022-11-27T17:45:01.959 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 10/138 2022-11-27T17:45:01.959 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 11/138 2022-11-27T17:45:01.959 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 12/138 2022-11-27T17:45:01.959 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 13/138 2022-11-27T17:45:01.960 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_6 14/138 2022-11-27T17:45:01.960 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 15/138 2022-11-27T17:45:01.960 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 16/138 2022-11-27T17:45:01.960 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 17/138 2022-11-27T17:45:01.961 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_ 18/138 2022-11-27T17:45:01.961 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 19/138 2022-11-27T17:45:01.961 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8 20/138 2022-11-27T17:45:01.961 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x 21/138 2022-11-27T17:45:01.962 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 22/138 2022-11-27T17:45:01.962 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 23/138 2022-11-27T17:45:01.962 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 24/138 2022-11-27T17:45:01.962 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 25/138 2022-11-27T17:45:01.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 26/138 2022-11-27T17:45:01.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 27/138 2022-11-27T17:45:01.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 28/138 2022-11-27T17:45:01.963 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.e 29/138 2022-11-27T17:45:01.964 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noar 30/138 2022-11-27T17:45:01.976 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.no 31/138 2022-11-27T17:45:01.976 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-594.g697b 32/138 2022-11-27T17:45:01.977 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8 33/138 2022-11-27T17:45:01.977 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 34/138 2022-11-27T17:45:01.977 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el 35/138 2022-11-27T17:45:01.977 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 36/138 2022-11-27T17:45:01.978 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:18.0.0-594.g697b0d99.el8.noarch 37/138 2022-11-27T17:45:01.978 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 38/138 2022-11-27T17:45:01.978 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/138 2022-11-27T17:45:01.978 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/138 2022-11-27T17:45:01.979 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/138 2022-11-27T17:45:01.979 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/138 2022-11-27T17:45:01.979 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/138 2022-11-27T17:45:01.979 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/138 2022-11-27T17:45:01.980 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/138 2022-11-27T17:45:01.980 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/138 2022-11-27T17:45:01.980 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/138 2022-11-27T17:45:01.980 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/138 2022-11-27T17:45:01.981 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/138 2022-11-27T17:45:01.981 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/138 2022-11-27T17:45:01.981 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/138 2022-11-27T17:45:01.981 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/138 2022-11-27T17:45:01.982 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/138 2022-11-27T17:45:01.982 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/138 2022-11-27T17:45:01.982 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/138 2022-11-27T17:45:01.982 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/138 2022-11-27T17:45:01.983 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/138 2022-11-27T17:45:01.983 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/138 2022-11-27T17:45:01.983 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/138 2022-11-27T17:45:01.983 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/138 2022-11-27T17:45:01.984 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/138 2022-11-27T17:45:01.984 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/138 2022-11-27T17:45:01.984 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/138 2022-11-27T17:45:01.985 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/138 2022-11-27T17:45:01.985 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/138 2022-11-27T17:45:01.986 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/138 2022-11-27T17:45:01.986 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/138 2022-11-27T17:45:01.987 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/138 2022-11-27T17:45:01.987 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/138 2022-11-27T17:45:01.988 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/138 2022-11-27T17:45:01.988 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/138 2022-11-27T17:45:01.989 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 72/138 2022-11-27T17:45:01.989 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 73/138 2022-11-27T17:45:01.990 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 74/138 2022-11-27T17:45:01.990 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/138 2022-11-27T17:45:01.990 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 76/138 2022-11-27T17:45:01.991 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 77/138 2022-11-27T17:45:01.991 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 78/138 2022-11-27T17:45:01.992 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 79/138 2022-11-27T17:45:01.992 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 80/138 2022-11-27T17:45:01.993 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 81/138 2022-11-27T17:45:01.993 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 82/138 2022-11-27T17:45:01.993 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 83/138 2022-11-27T17:45:01.994 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 84/138 2022-11-27T17:45:01.994 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/138 2022-11-27T17:45:01.995 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 86/138 2022-11-27T17:45:01.995 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/138 2022-11-27T17:45:01.996 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 88/138 2022-11-27T17:45:01.996 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 89/138 2022-11-27T17:45:01.996 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 90/138 2022-11-27T17:45:01.996 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 91/138 2022-11-27T17:45:01.997 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 92/138 2022-11-27T17:45:01.997 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 93/138 2022-11-27T17:45:01.997 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 94/138 2022-11-27T17:45:01.997 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 95/138 2022-11-27T17:45:01.997 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 96/138 2022-11-27T17:45:01.998 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/138 2022-11-27T17:45:01.998 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 98/138 2022-11-27T17:45:01.998 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.0.4-10.el8.x86_64 99/138 2022-11-27T17:45:01.998 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 100/138 2022-11-27T17:45:01.999 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 101/138 2022-11-27T17:45:01.999 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 102/138 2022-11-27T17:45:01.999 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 103/138 2022-11-27T17:45:01.999 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 104/138 2022-11-27T17:45:02.000 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 105/138 2022-11-27T17:45:02.000 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 106/138 2022-11-27T17:45:02.000 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 107/138 2022-11-27T17:45:02.001 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/138 2022-11-27T17:45:02.001 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 109/138 2022-11-27T17:45:02.001 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 110/138 2022-11-27T17:45:02.001 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 111/138 2022-11-27T17:45:02.001 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/138 2022-11-27T17:45:02.002 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/138 2022-11-27T17:45:02.002 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 114/138 2022-11-27T17:45:02.002 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.1-9.el8.x86_64 115/138 2022-11-27T17:45:02.002 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 116/138 2022-11-27T17:45:02.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-3.el8.x86_64 117/138 2022-11-27T17:45:02.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 118/138 2022-11-27T17:45:02.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.1-7.el8.noarch 119/138 2022-11-27T17:45:02.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 120/138 2022-11-27T17:45:02.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 121/138 2022-11-27T17:45:02.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 122/138 2022-11-27T17:45:02.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpq-13.5-1.el8.x86_64 123/138 2022-11-27T17:45:02.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 124/138 2022-11-27T17:45:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 125/138 2022-11-27T17:45:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 126/138 2022-11-27T17:45:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 127/138 2022-11-27T17:45:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 128/138 2022-11-27T17:45:02.006 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:18.0.0-594.g697b0d99.el8.x86_64 129/138 2022-11-27T17:45:02.006 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/138 2022-11-27T17:45:02.006 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 131/138 2022-11-27T17:45:02.006 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/138 2022-11-27T17:45:02.007 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/138 2022-11-27T17:45:02.007 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/138 2022-11-27T17:45:02.007 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/138 2022-11-27T17:45:02.007 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/138 2022-11-27T17:45:02.008 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 137/138 2022-11-27T17:45:03.886 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 138/138 2022-11-27T17:45:03.886 INFO:teuthology.orchestra.run.smithi119.stdout:Installed products updated. 2022-11-27T17:45:04.161 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:04.161 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2022-11-27T17:45:04.162 INFO:teuthology.orchestra.run.smithi119.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2022-11-27T17:45:04.162 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.162 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.163 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:45:04.163 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:45:04.164 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2022-11-27T17:45:04.164 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2022-11-27T17:45:04.165 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.1-7.el8.noarch 2022-11-27T17:45:04.165 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.0.4-10.el8.x86_64 2022-11-27T17:45:04.165 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.166 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.166 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.167 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.167 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.168 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.168 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.168 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.169 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.169 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.170 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.171 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.171 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.171 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.172 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.172 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.172 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.172 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.173 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.173 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.174 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:18.0.0-594.g697b0d99.el8.noarch 2022-11-27T17:45:04.174 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-11-27T17:45:04.174 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.1-9.el8.x86_64 2022-11-27T17:45:04.174 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.7-15.el8.x86_64 2022-11-27T17:45:04.174 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-11-27T17:45:04.175 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-3.el8.x86_64 2022-11-27T17:45:04.175 INFO:teuthology.orchestra.run.smithi119.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-11-27T17:45:04.175 INFO:teuthology.orchestra.run.smithi119.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-11-27T17:45:04.175 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-11-27T17:45:04.176 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.176 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.176 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.177 INFO:teuthology.orchestra.run.smithi119.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-11-27T17:45:04.177 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-11-27T17:45:04.178 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.2-3.el8.x86_64 2022-11-27T17:45:04.178 INFO:teuthology.orchestra.run.smithi119.stdout: libpq-13.5-1.el8.x86_64 2022-11-27T17:45:04.178 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-11-27T17:45:04.179 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-11-27T17:45:04.179 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.180 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.180 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-11-27T17:45:04.181 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.181 INFO:teuthology.orchestra.run.smithi119.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-11-27T17:45:04.182 INFO:teuthology.orchestra.run.smithi119.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-11-27T17:45:04.182 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-25.el8.x86_64 2022-11-27T17:45:04.182 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.3.1-3.el8.x86_64 2022-11-27T17:45:04.183 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-11-27T17:45:04.183 INFO:teuthology.orchestra.run.smithi119.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-11-27T17:45:04.183 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-11-27T17:45:04.183 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-11-27T17:45:04.184 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.15-3.el8.x86_64 2022-11-27T17:45:04.184 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-11-27T17:45:04.184 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-11-27T17:45:04.184 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-11-27T17:45:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-11-27T17:45:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-11-27T17:45:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:45:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:45:04.185 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-11-27T17:45:04.186 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-11-27T17:45:04.186 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.5.1-7.el8.noarch 2022-11-27T17:45:04.186 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-11-27T17:45:04.186 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-11-27T17:45:04.187 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-11-27T17:45:04.187 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.187 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.187 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.187 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-11-27T17:45:04.188 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-11-27T17:45:04.188 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-11-27T17:45:04.188 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-11-27T17:45:04.188 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-11-27T17:45:04.189 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-11-27T17:45:04.189 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-6.2-6.el8.noarch 2022-11-27T17:45:04.189 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-11-27T17:45:04.189 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-11-27T17:45:04.189 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-11-27T17:45:04.190 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-11-27T17:45:04.190 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-11-27T17:45:04.190 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-11-27T17:45:04.190 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-11-27T17:45:04.191 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.0.6-13.el8.noarch 2022-11-27T17:45:04.191 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-11-27T17:45:04.191 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-11-27T17:45:04.191 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-11-27T17:45:04.192 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-11-27T17:45:04.192 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-11-27T17:45:04.192 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-11-27T17:45:04.192 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-11-27T17:45:04.193 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-2.6-1.el8.noarch 2022-11-27T17:45:04.193 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-11-27T17:45:04.193 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-11-27T17:45:04.193 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-11-27T17:45:04.194 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-11-27T17:45:04.194 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.14-14.el8.noarch 2022-11-27T17:45:04.194 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2017.2-9.el8.noarch 2022-11-27T17:45:04.194 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.195 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.195 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-11-27T17:45:04.195 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-11-27T17:45:04.195 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.196 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.4.1-12.el8.noarch 2022-11-27T17:45:04.196 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.8-1.el8.noarch 2022-11-27T17:45:04.196 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-11-27T17:45:04.196 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-11-27T17:45:04.196 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-11-27T17:45:04.197 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-11-27T17:45:04.197 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-11-27T17:45:04.197 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-11-27T17:45:04.197 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-11-27T17:45:04.198 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-11-27T17:45:04.198 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-11-27T17:45:04.198 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-11-27T17:45:04.198 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-11-27T17:45:04.198 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.199 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.199 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:18.0.0-594.g697b0d99.el8.x86_64 2022-11-27T17:45:04.199 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-1.el8.x86_64 2022-11-27T17:45:04.199 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.13.0-2.el8.x86_64 2022-11-27T17:45:04.200 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-43.el8.x86_64 2022-11-27T17:45:04.200 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-11-27T17:45:04.200 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-11-27T17:45:04.201 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-11-27T17:45:04.201 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:04.201 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2022-11-27T17:45:04.366 DEBUG:teuthology.parallel:result is None 2022-11-27T17:45:04.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2022-11-27T17:45:04.433 INFO:teuthology.orchestra.run.smithi099.stdout:RedHatEnterprise 2022-11-27T17:45:04.435 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:45:04.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-27T17:45:04.600 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-11-27T17:45:04.625 INFO:teuthology.orchestra.run.smithi099.stdout:18.0.0-594.g697b0d99.el8 2022-11-27T17:45:04.627 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-594.g697b0d99.el8 2022-11-27T17:45:04.627 INFO:teuthology.task.install:The correct ceph version 18.0.0-594.g697b0d99 is installed. 2022-11-27T17:45:04.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo lsb_release -is 2022-11-27T17:45:04.685 INFO:teuthology.orchestra.run.smithi119.stdout:RedHatEnterprise 2022-11-27T17:45:04.686 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-11-27T17:45:04.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=697b0d99d9a52df92b20c8e053dcd46c75f364a9 2022-11-27T17:45:04.840 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-11-27T17:45:04.864 INFO:teuthology.orchestra.run.smithi119.stdout:18.0.0-594.g697b0d99.el8 2022-11-27T17:45:04.864 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-594.g697b0d99.el8 2022-11-27T17:45:04.865 INFO:teuthology.task.install:The correct ceph version 18.0.0-594.g697b0d99 is installed. 2022-11-27T17:45:04.865 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-11-27T17:45:04.866 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:04.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-11-27T17:45:04.898 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:04.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-11-27T17:45:04.931 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-11-27T17:45:04.931 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:04.932 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2022-11-27T17:45:04.967 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-11-27T17:45:05.036 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:05.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2022-11-27T17:45:05.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-11-27T17:45:05.132 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-11-27T17:45:05.133 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:05.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2022-11-27T17:45:05.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-11-27T17:45:05.229 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:05.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2022-11-27T17:45:05.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-11-27T17:45:05.327 INFO:teuthology.run_tasks:Running task ceph... 2022-11-27T17:45:05.417 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-11-27T17:45:05.418 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2022-11-27T17:45:05.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2022-11-27T17:45:05.447 INFO:tasks.ceph:Disabling ceph logrotate... 2022-11-27T17:45:05.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2022-11-27T17:45:05.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2022-11-27T17:45:05.515 INFO:tasks.ceph:Creating extra log directories... 2022-11-27T17:45:05.515 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-11-27T17:45:05.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-11-27T17:45:05.592 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-11-27T17:45:05.593 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-11-27T17:45:05.593 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-11-27T17:45:05.593 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:05.594 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-11-27T17:45:05.636 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:05.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:05.777 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:05.778 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-11-27T17:45:05.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:05.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:05.946 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-11-27T17:45:05.947 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'RECENT_CRASH'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-11-27T17:45:05.947 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi/7088721', 'branch': 'main', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '7088721', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-11-23_17:24:26-fs-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'RECENT_CRASH'], 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'dry_run': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_697b0d99d9a52df92b20c8e053dcd46c75f364a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi099.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAXQcBt9Pf4a4QsDxb/ldpDGiknTr5vOmmVVCzZWbXpmgIEtPEo5BUZ4ovE/y067yVpV1fTWLztHqtipJtdWedo=', 'smithi119.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEk6lqfLAicZY/cjf7eoGecbzwL+Frnvws+9uajk15RFsc1WaF9XwyBdeFdT0JY9zL1bEmFAMYoH1mBwL35icxy05ODKWD9m6oTu6Fb/bGeQHOtklmI373ymhJ8Vce8hoLRaQyIeJKKvusGxlkhaAszcbFtR+oBd11TqyuGe7IzO7254H0xKNjqbn7sc+oOK/3q6Bf8Xwzp6gSLzrlLVWxJBb1g/tS/GwuxX8Eu5kIX+oLSpPdwQrtZY2YYrqsjtV9ta3MonN5JlnvwQsa6NIrcynmlf5a0ri6XX+wYT2jZFKmJepz+NMXkyX9PksEfynIW4Vk2qBzP7zIeqJWXuTdew7f5lknTJvkXNqCplqVYqiAw3hijn2ajl5ycjJKvcbMWeFSCSE2sSmHHuf4ywCDQiukBO+GaWq2OI1butKcSPjotXADA46DHV9lTqV1pTopj9ALnyf/byQc3PucAUKJWspGhd3d6uD/xI2g47o4qWcLiNeWVgP9KfqC5IlGcFU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '697b0d99d9a52df92b20c8e053dcd46c75f364a9'}}, {'ceph': None}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_snapshots']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '4da97cf64e542f347ec47b7bdbe5eca99759f9b7', 'timestamp': '2022-11-23_17:24:26', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.20399'} 2022-11-27T17:45:05.948 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-11-27T17:45:05.951 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-11-27T17:45:05.996 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2022-11-27T17:45:06.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2022-11-27T17:45:06.064 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:06.065 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2022-11-27T17:45:06.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-11-27T17:45:06.091 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2022-11-27T17:45:06.152 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-11-27T17:45:06.152 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:06.152 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 367822 Links: 1 2022-11-27T17:45:06.153 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:06.153 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:06.153 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-11-27 17:44:51.985862756 +0000 2022-11-27T17:45:06.153 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-11-27 17:42:06.496058090 +0000 2022-11-27T17:45:06.154 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-11-27 17:42:06.496058090 +0000 2022-11-27T17:45:06.154 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-11-27T17:45:06.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-11-27T17:45:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-11-27T17:45:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-11-27T17:45:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000255789 s, 2.0 MB/s 2022-11-27T17:45:06.219 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-11-27T17:45:06.277 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2022-11-27T17:45:06.334 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-11-27T17:45:06.335 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:06.335 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 360788 Links: 1 2022-11-27T17:45:06.335 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:06.336 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:06.336 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-11-27 17:44:51.985862756 +0000 2022-11-27T17:45:06.336 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-11-27 17:42:07.018045529 +0000 2022-11-27T17:45:06.336 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-11-27 17:42:07.018045529 +0000 2022-11-27T17:45:06.336 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-11-27T17:45:06.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-11-27T17:45:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-11-27T17:45:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-11-27T17:45:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000211794 s, 2.4 MB/s 2022-11-27T17:45:06.402 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-11-27T17:45:06.461 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2022-11-27T17:45:06.518 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-11-27T17:45:06.518 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:06.519 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 359080 Links: 1 2022-11-27T17:45:06.519 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:06.519 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:06.519 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-11-27 17:44:51.986862732 +0000 2022-11-27T17:45:06.519 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-11-27 17:42:07.548032776 +0000 2022-11-27T17:45:06.520 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-11-27 17:42:07.548032776 +0000 2022-11-27T17:45:06.520 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-11-27T17:45:06.520 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-11-27T17:45:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-11-27T17:45:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-11-27T17:45:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000200845 s, 2.5 MB/s 2022-11-27T17:45:06.585 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-11-27T17:45:06.644 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2022-11-27T17:45:06.700 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-11-27T17:45:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:06.701 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 365371 Links: 1 2022-11-27T17:45:06.701 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2022-11-27 17:44:51.986862732 +0000 2022-11-27T17:45:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2022-11-27 17:42:08.096019589 +0000 2022-11-27T17:45:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2022-11-27 17:42:08.096019589 +0000 2022-11-27T17:45:06.703 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2022-11-27T17:45:06.703 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-11-27T17:45:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2022-11-27T17:45:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2022-11-27T17:45:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000161774 s, 3.2 MB/s 2022-11-27T17:45:06.767 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-11-27T17:45:06.826 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:06.827 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:06.828 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2022-11-27T17:45:06.848 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-11-27T17:45:06.849 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2022-11-27T17:45:06.908 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-11-27T17:45:06.908 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:06.909 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 358369 Links: 1 2022-11-27T17:45:06.909 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:06.910 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:06.910 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-11-27 17:45:01.635461325 +0000 2022-11-27T17:45:06.911 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-11-27 17:41:59.918211475 +0000 2022-11-27T17:45:06.911 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-11-27 17:41:59.918211475 +0000 2022-11-27T17:45:06.913 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-11-27T17:45:06.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-11-27T17:45:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-11-27T17:45:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-11-27T17:45:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000244275 s, 2.1 MB/s 2022-11-27T17:45:06.978 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-11-27T17:45:07.037 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2022-11-27T17:45:07.095 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-11-27T17:45:07.095 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:07.096 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 364825 Links: 1 2022-11-27T17:45:07.096 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:07.096 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:07.096 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-11-27 17:45:01.635461325 +0000 2022-11-27T17:45:07.097 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-11-27 17:42:00.465198334 +0000 2022-11-27T17:45:07.097 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-11-27 17:42:00.465198334 +0000 2022-11-27T17:45:07.097 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-11-27T17:45:07.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-11-27T17:45:07.162 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-11-27T17:45:07.163 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-11-27T17:45:07.163 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000245187 s, 2.1 MB/s 2022-11-27T17:45:07.165 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-11-27T17:45:07.224 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2022-11-27T17:45:07.283 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-11-27T17:45:07.283 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:07.283 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 364900 Links: 1 2022-11-27T17:45:07.284 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:07.284 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:07.284 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-11-27 17:45:01.635461325 +0000 2022-11-27T17:45:07.284 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-11-27 17:42:00.994185627 +0000 2022-11-27T17:45:07.285 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-11-27 17:42:00.994185627 +0000 2022-11-27T17:45:07.285 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-11-27T17:45:07.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-11-27T17:45:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-11-27T17:45:07.350 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-11-27T17:45:07.350 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000173449 s, 3.0 MB/s 2022-11-27T17:45:07.351 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-11-27T17:45:07.411 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2022-11-27T17:45:07.470 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-11-27T17:45:07.470 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-11-27T17:45:07.471 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 368131 Links: 1 2022-11-27T17:45:07.471 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-11-27T17:45:07.472 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2022-11-27T17:45:07.472 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2022-11-27 17:45:01.635461325 +0000 2022-11-27T17:45:07.473 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2022-11-27 17:42:01.526172846 +0000 2022-11-27T17:45:07.473 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2022-11-27 17:42:01.526172846 +0000 2022-11-27T17:45:07.474 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2022-11-27T17:45:07.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-11-27T17:45:07.535 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2022-11-27T17:45:07.535 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2022-11-27T17:45:07.536 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00023733 s, 2.2 MB/s 2022-11-27T17:45:07.537 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-11-27T17:45:07.596 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:07.596 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-11-27T17:45:07.597 INFO:tasks.ceph:Generating config... 2022-11-27T17:45:07.599 INFO:tasks.ceph:[client] client mount timeout = 600 2022-11-27T17:45:07.600 INFO:tasks.ceph:[client] debug client = 20 2022-11-27T17:45:07.600 INFO:tasks.ceph:[client] debug ms = 1 2022-11-27T17:45:07.600 INFO:tasks.ceph:[client] pid file = 2022-11-27T17:45:07.601 INFO:tasks.ceph:[client] rados mon op timeout = 900 2022-11-27T17:45:07.601 INFO:tasks.ceph:[client] rados osd op timeout = 900 2022-11-27T17:45:07.601 INFO:tasks.ceph:[global] ms die on skipped message = False 2022-11-27T17:45:07.602 INFO:tasks.ceph:[mds] debug mds = 20 2022-11-27T17:45:07.602 INFO:tasks.ceph:[mds] debug mds balancer = 20 2022-11-27T17:45:07.602 INFO:tasks.ceph:[mds] debug ms = 1 2022-11-27T17:45:07.603 INFO:tasks.ceph:[mds] mds debug frag = True 2022-11-27T17:45:07.603 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-11-27T17:45:07.603 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-11-27T17:45:07.604 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-11-27T17:45:07.604 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-11-27T17:45:07.604 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2022-11-27T17:45:07.605 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2022-11-27T17:45:07.605 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-11-27T17:45:07.605 INFO:tasks.ceph:[mgr] debug ms = 1 2022-11-27T17:45:07.606 INFO:tasks.ceph:[mon] debug mon = 20 2022-11-27T17:45:07.606 INFO:tasks.ceph:[mon] debug ms = 1 2022-11-27T17:45:07.606 INFO:tasks.ceph:[mon] debug paxos = 20 2022-11-27T17:45:07.606 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-11-27T17:45:07.607 INFO:tasks.ceph:[osd] bdev async discard = True 2022-11-27T17:45:07.607 INFO:tasks.ceph:[osd] bdev enable discard = True 2022-11-27T17:45:07.607 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2022-11-27T17:45:07.608 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-11-27T17:45:07.609 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-11-27T17:45:07.609 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2022-11-27T17:45:07.610 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2022-11-27T17:45:07.610 INFO:tasks.ceph:[osd] debug ms = 1 2022-11-27T17:45:07.611 INFO:tasks.ceph:[osd] debug osd = 20 2022-11-27T17:45:07.612 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2022-11-27T17:45:07.612 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-11-27T17:45:07.613 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-11-27T17:45:07.613 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-11-27T17:45:07.614 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-11-27T17:45:07.614 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-11-27T17:45:07.615 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-11-27T17:45:07.616 INFO:tasks.ceph:Setting up mon.a... 2022-11-27T17:45:07.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-11-27T17:45:07.684 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.keyring 2022-11-27T17:45:07.688 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-11-27T17:45:07.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-11-27T17:45:07.763 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.99'), ('mon.b', '172.21.15.119'), ('mon.c', '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]')] 2022-11-27T17:45:07.765 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2022-11-27T17:45:07.766 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:07.767 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-11-27T17:45:07.822 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --add b 172.21.15.119 --addv c '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:07.909 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:07.910 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid 03183fc1-5cf9-40d3-b24e-1fc3808ae993 2022-11-27T17:45:07.910 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = pacific 2022-11-27T17:45:07.910 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2022-11-27T17:45:07.911 INFO:teuthology.orchestra.run.smithi099.stdout:fsid 03183fc1-5cf9-40d3-b24e-1fc3808ae993 2022-11-27T17:45:07.911 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2022-11-27T17:45:07.908867+0000 2022-11-27T17:45:07.912 INFO:teuthology.orchestra.run.smithi099.stdout:created 2022-11-27T17:45:07.908867+0000 2022-11-27T17:45:07.912 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 16 (pacific) 2022-11-27T17:45:07.913 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 1 2022-11-27T17:45:07.913 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2022-11-27T17:45:07.914 INFO:teuthology.orchestra.run.smithi099.stdout:1: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2022-11-27T17:45:07.914 INFO:teuthology.orchestra.run.smithi099.stdout:2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2022-11-27T17:45:07.914 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-11-27T17:45:07.917 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-11-27T17:45:07.934 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 03183fc1-5cf9-40d3-b24e-1fc3808ae993... 2022-11-27T17:45:07.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-11-27T17:45:07.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-11-27T17:45:08.020 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2022-11-27T17:45:08.021 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2022-11-27T17:45:08.021 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-11-27T17:45:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2022-11-27T17:45:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2022-11-27T17:45:08.022 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2022-11-27T17:45:08.023 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2022-11-27T17:45:08.024 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2022-11-27T17:45:08.025 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2022-11-27T17:45:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2022-11-27T17:45:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.026 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2022-11-27T17:45:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2022-11-27T17:45:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2022-11-27T17:45:08.027 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2022-11-27T17:45:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2022-11-27T17:45:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2022-11-27T17:45:08.028 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2022-11-27T17:45:08.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2022-11-27T17:45:08.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2022-11-27T17:45:08.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2022-11-27T17:45:08.029 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2022-11-27T17:45:08.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2022-11-27T17:45:08.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-11-27T17:45:08.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2022-11-27T17:45:08.030 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2022-11-27T17:45:08.030 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.031 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-11-27T17:45:08.031 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.031 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2022-11-27T17:45:08.032 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.032 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2022-11-27T17:45:08.032 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.032 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2022-11-27T17:45:08.033 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2022-11-27T17:45:08.033 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2022-11-27T17:45:08.033 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = "172.21.15.99,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2022-11-27T17:45:08.034 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on skipped message = False 2022-11-27T17:45:08.034 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = 03183fc1-5cf9-40d3-b24e-1fc3808ae993 2022-11-27T17:45:08.034 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.035 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2022-11-27T17:45:08.035 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2022-11-27T17:45:08.036 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.036 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2022-11-27T17:45:08.037 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2022-11-27T17:45:08.037 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.037 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2022-11-27T17:45:08.038 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2022-11-27T17:45:08.038 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.038 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2022-11-27T17:45:08.039 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2022-11-27T17:45:08.039 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2022-11-27T17:45:08.040 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.040 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2022-11-27T17:45:08.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2022-11-27T17:45:08.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.041 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2022-11-27T17:45:08.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.042 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2022-11-27T17:45:08.043 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.043 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2022-11-27T17:45:08.044 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.044 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2022-11-27T17:45:08.044 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2022-11-27T17:45:08.045 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.045 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2022-11-27T17:45:08.045 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2022-11-27T17:45:08.046 INFO:teuthology.orchestra.run.smithi099.stdout: bdev async discard = True 2022-11-27T17:45:08.046 INFO:teuthology.orchestra.run.smithi099.stdout: bdev enable discard = True 2022-11-27T17:45:08.046 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore allocator = bitmap 2022-11-27T17:45:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore block size = 96636764160 2022-11-27T17:45:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore fsck on mount = True 2022-11-27T17:45:08.047 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluefs = 1/20 2022-11-27T17:45:08.048 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluestore = 1/20 2022-11-27T17:45:08.048 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-11-27T17:45:08.048 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2022-11-27T17:45:08.049 INFO:teuthology.orchestra.run.smithi099.stdout: debug rocksdb = 4/10 2022-11-27T17:45:08.049 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd backfillfull_ratio = 0.85 2022-11-27T17:45:08.049 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd full ratio = 0.9 2022-11-27T17:45:08.049 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd nearfull ratio = 0.8 2022-11-27T17:45:08.050 INFO:teuthology.orchestra.run.smithi099.stdout: osd failsafe full ratio = 0.95 2022-11-27T17:45:08.050 INFO:teuthology.orchestra.run.smithi099.stdout: osd objectstore = bluestore 2022-11-27T17:45:08.050 INFO:teuthology.orchestra.run.smithi099.stdout: osd op complaint time = 180 2022-11-27T17:45:08.050 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.051 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2022-11-27T17:45:08.051 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-11-27T17:45:08.051 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2022-11-27T17:45:08.052 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-11-27T17:45:08.052 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-11-27T17:45:08.052 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-11-27T17:45:08.052 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-11-27T17:45:08.052 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2022-11-27T17:45:08.053 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.053 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2022-11-27T17:45:08.053 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-11-27T17:45:08.054 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2022-11-27T17:45:08.054 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2022-11-27T17:45:08.055 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2022-11-27T17:45:08.055 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2022-11-27T17:45:08.056 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2022-11-27T17:45:08.056 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2022-11-27T17:45:08.056 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2022-11-27T17:45:08.057 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2022-11-27T17:45:08.057 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2022-11-27T17:45:08.058 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.058 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-11-27T17:45:08.059 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2022-11-27T17:45:08.059 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2022-11-27T17:45:08.060 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.060 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2022-11-27T17:45:08.060 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-11-27T17:45:08.061 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-11-27T17:45:08.061 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.062 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2022-11-27T17:45:08.062 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2022-11-27T17:45:08.062 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.063 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2022-11-27T17:45:08.063 INFO:teuthology.orchestra.run.smithi099.stdout: mon op complaint time = 120 2022-11-27T17:45:08.064 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:08.064 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2022-11-27T17:45:08.064 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2022-11-27T17:45:08.065 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2022-11-27T17:45:08.065 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2022-11-27T17:45:08.066 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-11-27T17:45:08.066 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-11-27T17:45:08.066 INFO:teuthology.orchestra.run.smithi099.stdout: client mount timeout = 600 2022-11-27T17:45:08.066 INFO:teuthology.orchestra.run.smithi099.stdout: debug client = 20 2022-11-27T17:45:08.066 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-11-27T17:45:08.067 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = "" 2022-11-27T17:45:08.067 INFO:teuthology.orchestra.run.smithi099.stdout: rados mon op timeout = 900 2022-11-27T17:45:08.067 INFO:teuthology.orchestra.run.smithi099.stdout: rados osd op timeout = 900 2022-11-27T17:45:08.067 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2022-11-27T17:45:08.068 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.b] 2022-11-27T17:45:08.068 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.c] 2022-11-27T17:45:08.068 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.a] 2022-11-27T17:45:08.069 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.c] 2022-11-27T17:45:08.069 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.b] 2022-11-27T17:45:08.069 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.d] 2022-11-27T17:45:08.069 INFO:teuthology.orchestra.run.smithi099.stdout:[mds] 2022-11-27T17:45:08.069 INFO:teuthology.orchestra.run.smithi099.stdout: debug mds = 20 2022-11-27T17:45:08.070 INFO:teuthology.orchestra.run.smithi099.stdout: debug mds balancer = 20 2022-11-27T17:45:08.070 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2022-11-27T17:45:08.070 INFO:teuthology.orchestra.run.smithi099.stdout: mds debug frag = True 2022-11-27T17:45:08.070 INFO:teuthology.orchestra.run.smithi099.stdout: mds debug scatterstat = True 2022-11-27T17:45:08.071 INFO:teuthology.orchestra.run.smithi099.stdout: mds op complaint time = 180 2022-11-27T17:45:08.071 INFO:teuthology.orchestra.run.smithi099.stdout: mds verify scatter = True 2022-11-27T17:45:08.071 INFO:teuthology.orchestra.run.smithi099.stdout: osd op complaint time = 180 2022-11-27T17:45:08.071 INFO:teuthology.orchestra.run.smithi099.stdout: rados mon op timeout = 900 2022-11-27T17:45:08.072 INFO:teuthology.orchestra.run.smithi099.stdout: rados osd op timeout = 900 2022-11-27T17:45:08.074 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2022-11-27T17:45:08.074 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2022-11-27T17:45:08.074 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-11-27T17:45:08.075 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2022-11-27T17:45:08.075 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.075 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2022-11-27T17:45:08.075 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.076 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2022-11-27T17:45:08.076 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.076 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2022-11-27T17:45:08.076 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2022-11-27T17:45:08.077 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.077 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2022-11-27T17:45:08.077 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2022-11-27T17:45:08.077 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.078 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2022-11-27T17:45:08.078 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2022-11-27T17:45:08.078 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.079 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2022-11-27T17:45:08.079 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2022-11-27T17:45:08.079 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.080 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2022-11-27T17:45:08.080 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.080 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2022-11-27T17:45:08.081 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2022-11-27T17:45:08.081 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2022-11-27T17:45:08.081 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2022-11-27T17:45:08.081 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2022-11-27T17:45:08.082 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2022-11-27T17:45:08.082 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2022-11-27T17:45:08.082 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-11-27T17:45:08.082 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2022-11-27T17:45:08.083 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2022-11-27T17:45:08.083 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.084 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-11-27T17:45:08.084 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.084 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2022-11-27T17:45:08.085 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.085 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2022-11-27T17:45:08.085 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.086 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2022-11-27T17:45:08.086 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2022-11-27T17:45:08.086 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2022-11-27T17:45:08.086 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.99,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2022-11-27T17:45:08.086 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on skipped message = False 2022-11-27T17:45:08.087 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 03183fc1-5cf9-40d3-b24e-1fc3808ae993 2022-11-27T17:45:08.087 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.087 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2022-11-27T17:45:08.088 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2022-11-27T17:45:08.088 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.088 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2022-11-27T17:45:08.088 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2022-11-27T17:45:08.089 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.089 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2022-11-27T17:45:08.089 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2022-11-27T17:45:08.089 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.090 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2022-11-27T17:45:08.090 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2022-11-27T17:45:08.090 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2022-11-27T17:45:08.090 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.090 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2022-11-27T17:45:08.091 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2022-11-27T17:45:08.091 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.091 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2022-11-27T17:45:08.091 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.092 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2022-11-27T17:45:08.092 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.092 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2022-11-27T17:45:08.092 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.093 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2022-11-27T17:45:08.093 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2022-11-27T17:45:08.093 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.093 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2022-11-27T17:45:08.093 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2022-11-27T17:45:08.094 INFO:teuthology.orchestra.run.smithi119.stdout: bdev async discard = True 2022-11-27T17:45:08.094 INFO:teuthology.orchestra.run.smithi119.stdout: bdev enable discard = True 2022-11-27T17:45:08.094 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore allocator = bitmap 2022-11-27T17:45:08.095 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2022-11-27T17:45:08.095 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2022-11-27T17:45:08.095 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 1/20 2022-11-27T17:45:08.095 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 1/20 2022-11-27T17:45:08.095 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2022-11-27T17:45:08.096 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2022-11-27T17:45:08.096 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 4/10 2022-11-27T17:45:08.096 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2022-11-27T17:45:08.096 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2022-11-27T17:45:08.097 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2022-11-27T17:45:08.097 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2022-11-27T17:45:08.097 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2022-11-27T17:45:08.097 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2022-11-27T17:45:08.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.098 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2022-11-27T17:45:08.098 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2022-11-27T17:45:08.098 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2022-11-27T17:45:08.098 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2022-11-27T17:45:08.099 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2022-11-27T17:45:08.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2022-11-27T17:45:08.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2022-11-27T17:45:08.099 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2022-11-27T17:45:08.099 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.100 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2022-11-27T17:45:08.100 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2022-11-27T17:45:08.100 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2022-11-27T17:45:08.100 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2022-11-27T17:45:08.101 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2022-11-27T17:45:08.101 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2022-11-27T17:45:08.101 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2022-11-27T17:45:08.101 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2022-11-27T17:45:08.101 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2022-11-27T17:45:08.102 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2022-11-27T17:45:08.102 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2022-11-27T17:45:08.102 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.102 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-11-27T17:45:08.103 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2022-11-27T17:45:08.103 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2022-11-27T17:45:08.103 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.103 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2022-11-27T17:45:08.103 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-11-27T17:45:08.104 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-11-27T17:45:08.104 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.104 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2022-11-27T17:45:08.104 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2022-11-27T17:45:08.105 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.105 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2022-11-27T17:45:08.105 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2022-11-27T17:45:08.105 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-11-27T17:45:08.106 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2022-11-27T17:45:08.106 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2022-11-27T17:45:08.106 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2022-11-27T17:45:08.106 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2022-11-27T17:45:08.106 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-11-27T17:45:08.107 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-11-27T17:45:08.107 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2022-11-27T17:45:08.107 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2022-11-27T17:45:08.107 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2022-11-27T17:45:08.108 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2022-11-27T17:45:08.108 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2022-11-27T17:45:08.108 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2022-11-27T17:45:08.108 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2022-11-27T17:45:08.109 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2022-11-27T17:45:08.109 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2022-11-27T17:45:08.109 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2022-11-27T17:45:08.109 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2022-11-27T17:45:08.110 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2022-11-27T17:45:08.110 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2022-11-27T17:45:08.110 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2022-11-27T17:45:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2022-11-27T17:45:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2022-11-27T17:45:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2022-11-27T17:45:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2022-11-27T17:45:08.111 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = True 2022-11-27T17:45:08.112 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2022-11-27T17:45:08.112 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = True 2022-11-27T17:45:08.112 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2022-11-27T17:45:08.112 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2022-11-27T17:45:08.113 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2022-11-27T17:45:08.113 INFO:tasks.ceph:Creating admin key on mon.a... 2022-11-27T17:45:08.114 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-11-27T17:45:08.171 INFO:tasks.ceph:Copying monmap to all nodes... 2022-11-27T17:45:08.172 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:08.173 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2022-11-27T17:45:08.193 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:08.194 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2022-11-27T17:45:08.249 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:45:08.250 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:08.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.keyring 2022-11-27T17:45:08.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-11-27T17:45:08.327 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:08.328 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:08.383 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:45:08.384 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:08.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2022-11-27T17:45:08.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-11-27T17:45:08.425 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:08.425 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:08.483 INFO:tasks.ceph:Setting up mon nodes... 2022-11-27T17:45:08.483 INFO:tasks.ceph:Setting up mgr nodes... 2022-11-27T17:45:08.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-11-27T17:45:08.537 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-11-27T17:45:08.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-11-27T17:45:08.607 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-11-27T17:45:08.609 INFO:tasks.ceph:Setting up mds nodes... 2022-11-27T17:45:08.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-11-27T17:45:08.663 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-11-27T17:45:08.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-11-27T17:45:08.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2022-11-27T17:45:08.785 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2022-11-27T17:45:08.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2022-11-27T17:45:08.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-11-27T17:45:08.871 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-11-27T17:45:08.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-11-27T17:45:08.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2022-11-27T17:45:08.993 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2022-11-27T17:45:08.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2022-11-27T17:45:09.023 INFO:tasks.ceph_client:Setting up client nodes... 2022-11-27T17:45:09.024 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-11-27T17:45:09.070 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-11-27T17:45:09.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2022-11-27T17:45:09.127 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.2.keyring 2022-11-27T17:45:09.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-11-27T17:45:09.184 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-11-27T17:45:09.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2022-11-27T17:45:09.280 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.3.keyring 2022-11-27T17:45:09.293 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-11-27T17:45:09.293 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-11-27T17:45:09.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-11-27T17:45:09.321 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:09.321 INFO:tasks.ceph:role: osd.0 2022-11-27T17:45:09.322 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:45:09.323 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-11-27T17:45:09.394 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:09.395 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:09.395 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:09.395 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:09.396 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:09.396 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:09.396 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:09.397 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:09.397 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:09.397 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:09.828 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-11-27T17:45:09.830 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-11-27T17:45:09.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-11-27T17:45:10.024 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-11-27T17:45:10.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-11-27T17:45:10.134 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:10.134 INFO:tasks.ceph:role: osd.1 2022-11-27T17:45:10.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:45:10.135 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-11-27T17:45:10.208 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:10.209 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:10.209 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:10.209 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:10.210 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:10.210 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:10.210 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:10.211 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:10.211 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:10.211 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:10.619 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-11-27T17:45:10.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-11-27T17:45:10.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-11-27T17:45:10.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-11-27T17:45:10.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-11-27T17:45:10.836 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:10.837 INFO:tasks.ceph:role: osd.2 2022-11-27T17:45:10.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:45:10.838 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-11-27T17:45:10.913 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:10.914 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:10.914 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:10.914 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:10.915 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:10.915 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:10.916 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:10.916 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:10.917 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:10.917 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:11.368 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-11-27T17:45:11.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-11-27T17:45:11.370 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-11-27T17:45:11.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-11-27T17:45:11.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-11-27T17:45:11.636 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:11.637 INFO:tasks.ceph:role: osd.3 2022-11-27T17:45:11.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com 2022-11-27T17:45:11.637 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-11-27T17:45:11.714 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:11.714 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:11.714 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:11.714 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:11.715 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:11.715 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:11.715 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:11.715 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:11.716 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:11.716 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:12.149 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2022-11-27T17:45:12.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2022-11-27T17:45:12.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-11-27T17:45:12.342 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-11-27T17:45:12.377 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.464+0000 7fa0aa92c5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-11-27T17:45:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.464+0000 7fa0aa92c5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-11-27T17:45:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.464+0000 7fa0aa92c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-27T17:45:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.464+0000 7fa0aa92c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-27T17:45:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.464+0000 7fa0aa92c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-11-27T17:45:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:12.465+0000 7fa0aa92c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-11-27T17:45:15.339 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-11-27T17:45:15.373 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.463+0000 7fc4ebfce5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-11-27T17:45:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.463+0000 7fc4ebfce5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-11-27T17:45:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.463+0000 7fc4ebfce5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-27T17:45:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.463+0000 7fc4ebfce5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-27T17:45:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.463+0000 7fc4ebfce5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-11-27T17:45:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:15.464+0000 7fc4ebfce5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-11-27T17:45:18.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-11-27T17:45:18.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.438+0000 7f02bb0755c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-11-27T17:45:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.438+0000 7f02bb0755c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-11-27T17:45:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.438+0000 7f02bb0755c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-27T17:45:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.438+0000 7f02bb0755c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-27T17:45:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.438+0000 7f02bb0755c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-11-27T17:45:18.443 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:18.439+0000 7f02bb0755c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-11-27T17:45:21.308 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-11-27T17:45:21.340 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.428+0000 7f056c82c5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-11-27T17:45:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.428+0000 7f056c82c5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-11-27T17:45:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.428+0000 7f056c82c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-11-27T17:45:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.428+0000 7f056c82c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-11-27T17:45:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.428+0000 7f056c82c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-11-27T17:45:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:21.429+0000 7f056c82c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2022-11-27T17:45:24.286 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-11-27T17:45:24.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-11-27T17:45:24.352 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:24.352 INFO:tasks.ceph:role: osd.4 2022-11-27T17:45:24.353 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:45:24.353 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-11-27T17:45:24.430 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:24.430 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:24.431 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:24.431 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:24.432 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:24.432 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:24.433 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:24.433 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:24.433 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:24.434 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:24.856 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2022-11-27T17:45:24.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2022-11-27T17:45:24.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-11-27T17:45:25.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-11-27T17:45:25.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-11-27T17:45:25.131 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:25.132 INFO:tasks.ceph:role: osd.5 2022-11-27T17:45:25.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:45:25.133 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-11-27T17:45:25.208 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:25.209 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:25.209 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:25.209 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:25.209 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:25.210 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:25.210 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:25.210 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:25.210 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:25.211 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:25.633 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2022-11-27T17:45:25.634 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2022-11-27T17:45:25.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-11-27T17:45:25.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-11-27T17:45:25.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-11-27T17:45:25.860 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:25.861 INFO:tasks.ceph:role: osd.6 2022-11-27T17:45:25.861 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:45:25.862 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-11-27T17:45:25.936 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:25.936 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:25.937 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:25.937 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:25.937 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:25.937 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:25.938 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:25.939 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:25.939 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:25.939 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:26.380 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2022-11-27T17:45:26.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2022-11-27T17:45:26.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-11-27T17:45:26.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-11-27T17:45:26.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-11-27T17:45:26.632 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-11-27T17:45:26.633 INFO:tasks.ceph:role: osd.7 2022-11-27T17:45:26.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2022-11-27T17:45:26.633 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-11-27T17:45:26.709 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-11-27T17:45:26.709 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2022-11-27T17:45:26.710 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-11-27T17:45:26.710 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=0 inobtcount=0 2022-11-27T17:45:26.711 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-11-27T17:45:26.711 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2022-11-27T17:45:26.711 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-11-27T17:45:26.711 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-11-27T17:45:26.712 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-11-27T17:45:26.712 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-11-27T17:45:27.161 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2022-11-27T17:45:27.164 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2022-11-27T17:45:27.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-11-27T17:45:27.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-11-27T17:45:27.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:27.488 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.486+0000 7fafc78a95c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-11-27T17:45:27.489 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.486+0000 7fafc78a95c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-11-27T17:45:27.489 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.486+0000 7fafc78a95c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-11-27T17:45:27.489 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.486+0000 7fafc78a95c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-11-27T17:45:27.490 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.486+0000 7fafc78a95c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-11-27T17:45:27.490 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:27.487+0000 7fafc78a95c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-11-27T17:45:30.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-11-27T17:45:30.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:30.478 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.475+0000 7fe452bba5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-11-27T17:45:30.479 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.475+0000 7fe452bba5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-11-27T17:45:30.479 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.475+0000 7fe452bba5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-11-27T17:45:30.480 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.475+0000 7fe452bba5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-11-27T17:45:30.480 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.475+0000 7fe452bba5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-11-27T17:45:30.480 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:30.476+0000 7fe452bba5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2022-11-27T17:45:33.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-11-27T17:45:33.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:33.474 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.471+0000 7f560058c5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-11-27T17:45:33.475 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.472+0000 7f560058c5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-11-27T17:45:33.475 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.472+0000 7f560058c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-11-27T17:45:33.475 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.472+0000 7f560058c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-11-27T17:45:33.475 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.472+0000 7f560058c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-11-27T17:45:33.476 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:33.472+0000 7f560058c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2022-11-27T17:45:35.949 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:35.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-11-27T17:45:36.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-11-27T17:45:36.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:36.464 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.461+0000 7fd5a6ee55c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-11-27T17:45:36.464 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.462+0000 7fd5a6ee55c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-11-27T17:45:36.465 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.462+0000 7fd5a6ee55c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-11-27T17:45:36.465 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.462+0000 7fd5a6ee55c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-11-27T17:45:36.465 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.462+0000 7fd5a6ee55c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-11-27T17:45:36.465 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:36.462+0000 7fd5a6ee55c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2022-11-27T17:45:39.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-11-27T17:45:39.359 INFO:tasks.ceph:Reading keys from all nodes... 2022-11-27T17:45:39.360 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.360 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2022-11-27T17:45:39.393 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.394 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2022-11-27T17:45:39.464 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2022-11-27T17:45:39.536 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2022-11-27T17:45:39.607 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.608 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2022-11-27T17:45:39.678 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2022-11-27T17:45:39.747 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:39.747 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2022-11-27T17:45:39.815 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:39.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2022-11-27T17:45:39.847 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:39.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2022-11-27T17:45:39.918 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:39.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2022-11-27T17:45:39.986 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:39.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2022-11-27T17:45:40.056 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:40.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2022-11-27T17:45:40.123 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:40.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2022-11-27T17:45:40.191 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:40.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2022-11-27T17:45:40.258 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:40.259 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2022-11-27T17:45:40.279 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:40.279 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2022-11-27T17:45:40.339 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:40.340 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2022-11-27T17:45:40.357 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:40.358 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2022-11-27T17:45:40.417 INFO:tasks.ceph:Adding keys to all mons... 2022-11-27T17:45:40.418 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/ceph.keyring 2022-11-27T17:45:40.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2022-11-27T17:45:40.460 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.y] 2022-11-27T17:45:40.460 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNjP0UEIBAAH2VKy3GTZYjGLeeWKcEQsg== 2022-11-27T17:45:40.461 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.a] 2022-11-27T17:45:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNjQ0KJJxAA8qZL48uh4ZhYgpUXsefIOQ== 2022-11-27T17:45:40.461 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.c] 2022-11-27T17:45:40.461 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNjhcDOLhAADTdjS9UmDg8fbN6jjJGlUA== 2022-11-27T17:45:40.462 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2022-11-27T17:45:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAoooNjqCTHGxAAO4wU3UU1GgJ5v/hQ+u0eUw== 2022-11-27T17:45:40.462 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2022-11-27T17:45:40.462 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQArooNjQ+WlGxAA2sJdjHhA7ZUjs/nCguzF7A== 2022-11-27T17:45:40.462 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2022-11-27T17:45:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAuooNjRgUqGhAArGYdOJQExwhg7x+/MfQQog== 2022-11-27T17:45:40.463 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.3] 2022-11-27T17:45:40.463 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAxooNjYzyRGRAAuym3b8sGa9dluGFoQKwroA== 2022-11-27T17:45:40.463 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2022-11-27T17:45:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNji48lJBAA7Bs+TU+BSNyQVw7mY+Z+/w== 2022-11-27T17:45:40.464 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.b] 2022-11-27T17:45:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNjpm7mMxAA2/U5q05Q2VP8DWa9cbj3Lw== 2022-11-27T17:45:40.464 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.d] 2022-11-27T17:45:40.464 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAkooNjbUYxOxAA0HhiUNeT0IjYPJQhUrJlwA== 2022-11-27T17:45:40.465 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.4] 2022-11-27T17:45:40.465 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQA3ooNjPLoQHRAAy0ZaZqmKr+vOkvuctaRo7A== 2022-11-27T17:45:40.465 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.5] 2022-11-27T17:45:40.465 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQA6ooNjN7lpHBAAQCyF6y9F+Oxe/eqY+rXnyQ== 2022-11-27T17:45:40.465 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.6] 2022-11-27T17:45:40.466 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQA9ooNjzz8yHBAAjSqj/9066iLLQ9txgLs08w== 2022-11-27T17:45:40.466 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.7] 2022-11-27T17:45:40.466 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBAooNj3aOZGxAAPSmE9CLPPHvGPdXorveduQ== 2022-11-27T17:45:40.466 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2022-11-27T17:45:40.466 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAlooNjpsUsBBAAemWWEgSrnqaIX7+2c0khfA== 2022-11-27T17:45:40.467 INFO:teuthology.orchestra.run.smithi099.stdout:[client.2] 2022-11-27T17:45:40.467 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAlooNjUcSVBxAA/eijk2WsgzT366gwuUI/OQ== 2022-11-27T17:45:40.467 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2022-11-27T17:45:40.467 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAlooNjFuD1ChAA6rxoFFoIU1zWSh5taMh+IA== 2022-11-27T17:45:40.468 INFO:teuthology.orchestra.run.smithi099.stdout:[client.3] 2022-11-27T17:45:40.468 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAlooNjkaqxEBAAcAxXFVGKjQFZ+cZNoie6ag== 2022-11-27T17:45:40.482 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2022-11-27T17:45:40.483 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNjP0UEIBAAH2VKy3GTZYjGLeeWKcEQsg== 2022-11-27T17:45:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2022-11-27T17:45:40.483 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNjQ0KJJxAA8qZL48uh4ZhYgpUXsefIOQ== 2022-11-27T17:45:40.484 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2022-11-27T17:45:40.484 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNjhcDOLhAADTdjS9UmDg8fbN6jjJGlUA== 2022-11-27T17:45:40.485 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2022-11-27T17:45:40.486 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAoooNjqCTHGxAAO4wU3UU1GgJ5v/hQ+u0eUw== 2022-11-27T17:45:40.486 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2022-11-27T17:45:40.486 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQArooNjQ+WlGxAA2sJdjHhA7ZUjs/nCguzF7A== 2022-11-27T17:45:40.486 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2022-11-27T17:45:40.487 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAuooNjRgUqGhAArGYdOJQExwhg7x+/MfQQog== 2022-11-27T17:45:40.487 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2022-11-27T17:45:40.487 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAxooNjYzyRGRAAuym3b8sGa9dluGFoQKwroA== 2022-11-27T17:45:40.487 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2022-11-27T17:45:40.488 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNji48lJBAA7Bs+TU+BSNyQVw7mY+Z+/w== 2022-11-27T17:45:40.488 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2022-11-27T17:45:40.488 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNjpm7mMxAA2/U5q05Q2VP8DWa9cbj3Lw== 2022-11-27T17:45:40.488 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2022-11-27T17:45:40.489 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAkooNjbUYxOxAA0HhiUNeT0IjYPJQhUrJlwA== 2022-11-27T17:45:40.489 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2022-11-27T17:45:40.489 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA3ooNjPLoQHRAAy0ZaZqmKr+vOkvuctaRo7A== 2022-11-27T17:45:40.489 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2022-11-27T17:45:40.490 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA6ooNjN7lpHBAAQCyF6y9F+Oxe/eqY+rXnyQ== 2022-11-27T17:45:40.490 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2022-11-27T17:45:40.490 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA9ooNjzz8yHBAAjSqj/9066iLLQ9txgLs08w== 2022-11-27T17:45:40.490 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2022-11-27T17:45:40.490 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBAooNj3aOZGxAAPSmE9CLPPHvGPdXorveduQ== 2022-11-27T17:45:40.491 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2022-11-27T17:45:40.491 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAlooNjpsUsBBAAemWWEgSrnqaIX7+2c0khfA== 2022-11-27T17:45:40.491 INFO:teuthology.orchestra.run.smithi119.stdout:[client.2] 2022-11-27T17:45:40.491 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAlooNjUcSVBxAA/eijk2WsgzT366gwuUI/OQ== 2022-11-27T17:45:40.492 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2022-11-27T17:45:40.492 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAlooNjFuD1ChAA6rxoFFoIU1zWSh5taMh+IA== 2022-11-27T17:45:40.492 INFO:teuthology.orchestra.run.smithi119.stdout:[client.3] 2022-11-27T17:45:40.492 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAlooNjkaqxEBAAcAxXFVGKjQFZ+cZNoie6ag== 2022-11-27T17:45:40.494 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-11-27T17:45:40.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-11-27T17:45:40.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:40.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:40.647 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:40.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:40.706 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.819 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:40.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-11-27T17:45:40.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-11-27T17:45:40.987 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:40.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:41.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:41.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-11-27T17:45:41.105 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.160 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.216 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.271 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-11-27T17:45:41.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.387 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.504 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-11-27T17:45:41.565 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-11-27T17:45:41.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-11-27T17:45:41.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-11-27T17:45:41.690 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-11-27T17:45:41.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-11-27T17:45:41.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-11-27T17:45:41.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-11-27T17:45:41.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-11-27T17:45:41.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-11-27T17:45:42.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-11-27T17:45:42.061 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:42.065 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-11-27T17:45:42.117 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-11-27T17:45:42.118 INFO:tasks.ceph.mon.a:Restarting daemon 2022-11-27T17:45:42.118 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-11-27T17:45:42.122 INFO:tasks.ceph.mon.a:Started 2022-11-27T17:45:42.123 INFO:tasks.ceph.mon.b:Restarting daemon 2022-11-27T17:45:42.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-11-27T17:45:42.161 INFO:tasks.ceph.mon.b:Started 2022-11-27T17:45:42.161 INFO:tasks.ceph.mon.c:Restarting daemon 2022-11-27T17:45:42.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-11-27T17:45:42.164 INFO:tasks.ceph.mon.c:Started 2022-11-27T17:45:42.164 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-11-27T17:45:42.164 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-11-27T17:45:42.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-11-27T17:45:42.167 INFO:tasks.ceph.mgr.y:Started 2022-11-27T17:45:42.168 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-11-27T17:45:42.168 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-11-27T17:45:42.170 INFO:tasks.ceph.mgr.x:Started 2022-11-27T17:45:42.171 INFO:tasks.ceph:Setting crush tunables to default 2022-11-27T17:45:42.171 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd crush tunables default 2022-11-27T17:45:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.292+0000 7f619f297700 1 Processor -- start 2022-11-27T17:45:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.292+0000 7f619f297700 1 -- start start 2022-11-27T17:45:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619f297700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 0x7f61980d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619f297700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980b1ca0 con 0x7f61980d0ae0 2022-11-27T17:45:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619f297700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61980cc270 con 0x7f61980b34d0 2022-11-27T17:45:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619f297700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61980cc830 con 0x7f61981598c0 2022-11-27T17:45:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619c832700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f61980d0ae0 0x7f61980b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52574/0 (socket says 172.21.15.99:52574) 2022-11-27T17:45:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619c832700 1 -- 172.21.15.99:0/695865312 learned_addr learned my addr 172.21.15.99:0/695865312 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:45:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 0x7f61980d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619d033700 1 -- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 msgr2=0x7f61980d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-11-27T17:45:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619d033700 1 -- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 msgr2=0x7f61980d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-11-27T17:45:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 0x7f61980d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-11-27T17:45:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.293+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 0x7f61980d05a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-11-27T17:45:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192088474 0 0) 0x7f61980b1ca0 con 0x7f61980d0ae0 2022-11-27T17:45:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c0035b0 con 0x7f61980d0ae0 2022-11-27T17:45:42.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2669416905 0 0) 0x7f617c0035b0 con 0x7f61980d0ae0 2022-11-27T17:45:42.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 msgr2=0x7f61980d05a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-11-27T17:45:42.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 --2- 172.21.15.99:0/695865312 >> v2:172.21.15.119:3300/0 conn(0x7f61981598c0 0x7f61980d05a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.487 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 >> v1:172.21.15.119:6789/0 conn(0x7f61980b34d0 legacy=0x7f619806d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980cc550 con 0x7f61980d0ae0 2022-11-27T17:45:42.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2447476370 0 0) 0x7f6188001670 con 0x7f61980d0ae0 2022-11-27T17:45:42.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 >> v1:172.21.15.99:6789/0 conn(0x7f61980d0ae0 legacy=0x7f61980b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.488 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f617c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.489 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f617c004850 0x7f617c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.489 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f617c005200 0x7f617c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.489 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980cc830 con 0x7f617c003a00 2022-11-27T17:45:42.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c00c190 con 0x7f617c004850 2022-11-27T17:45:42.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c00c350 con 0x7f617c005200 2022-11-27T17:45:42.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f617c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.490 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619c832700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f617c004850 0x7f617c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d834700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f617c005200 0x7f617c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 -- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f617c005200 msgr2=0x7f617c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f617c005200 0x7f617c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 -- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f617c004850 msgr2=0x7f617c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f617c004850 0x7f617c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d033700 1 -- 172.21.15.99:0/695865312 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0035b0 con 0x7f617c003a00 2022-11-27T17:45:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.480+0000 7f619d834700 1 -- 172.21.15.99:0/695865312 reap_dead start 2022-11-27T17:45:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619d033700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f617c003e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6194009860 tx=0x7f617c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb62b4198cebc04 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:42.492 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619400c070 con 0x7f617c003a00 2022-11-27T17:45:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194004020 con 0x7f617c003a00 2022-11-27T17:45:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f618ffff700 1 -- 172.21.15.99:0/695865312 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61940043b0 con 0x7f617c003a00 2022-11-27T17:45:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619f297700 1 -- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 msgr2=0x7f617c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619f297700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f617c003e10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6194009860 tx=0x7f617c005880 comp rx=0 tx=0).stop 2022-11-27T17:45:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619f297700 1 -- 172.21.15.99:0/695865312 shutdown_connections 2022-11-27T17:45:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619f297700 1 --2- 172.21.15.99:0/695865312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f617c003e10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.481+0000 7f619f297700 1 -- 172.21.15.99:0/695865312 >> 172.21.15.99:0/695865312 conn(0x7f6198071060 msgr2=0x7f6198071430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.482+0000 7f619f297700 1 -- 172.21.15.99:0/695865312 shutdown_connections 2022-11-27T17:45:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.482+0000 7f619f297700 1 -- 172.21.15.99:0/695865312 wait complete. 2022-11-27T17:45:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.483+0000 7f619f297700 1 Processor -- start 2022-11-27T17:45:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.483+0000 7f619f297700 1 -- start start 2022-11-27T17:45:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.483+0000 7f619f297700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.483+0000 7f619f297700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f619815c430 0x7f61981f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.483+0000 7f619f297700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f619815c8a0 0x7f61981f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619f297700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619815ad00 con 0x7f617c003a00 2022-11-27T17:45:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619f297700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619815a0f0 con 0x7f619815c8a0 2022-11-27T17:45:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619f297700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619815a3d0 con 0x7f619815c430 2022-11-27T17:45:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38342/0 (socket says 172.21.15.99:38342) 2022-11-27T17:45:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 -- 172.21.15.99:0/1550599382 learned_addr learned my addr 172.21.15.99:0/1550599382 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:45:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d834700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f619815c8a0 0x7f61981f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619c832700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f619815c430 0x7f61981f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 -- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f619815c430 msgr2=0x7f61981f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f619815c430 0x7f61981f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 -- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f619815c8a0 msgr2=0x7f61981f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f619815c8a0 0x7f61981f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 -- 172.21.15.99:0/1550599382 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619815a6b0 con 0x7f617c003a00 2022-11-27T17:45:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.484+0000 7f619d033700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6194000c00 tx=0x7f61940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61940045a0 con 0x7f617c003a00 2022-11-27T17:45:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194003d90 con 0x7f617c003a00 2022-11-27T17:45:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619815a990 con 0x7f617c003a00 2022-11-27T17:45:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194026680 con 0x7f617c003a00 2022-11-27T17:45:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61981eea20 con 0x7f617c003a00 2022-11-27T17:45:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f619402c020 con 0x7f617c003a00 2022-11-27T17:45:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.487+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f619402f780 con 0x7f617c003a00 2022-11-27T17:45:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.488+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6198073030 con 0x7f617c003a00 2022-11-27T17:45:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.490+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f6194017030 con 0x7f617c003a00 2022-11-27T17:45:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.565+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f61980affe0 con 0x7f617c003a00 2022-11-27T17:45:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.572+0000 7f618dffb700 1 -- 172.21.15.99:0/1550599382 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f619400f040 con 0x7f617c003a00 2022-11-27T17:45:42.576 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.573+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 msgr2=0x7f61981ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:42.576 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.573+0000 7f619f297700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6194000c00 tx=0x7f61940047c0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.576 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 shutdown_connections 2022-11-27T17:45:42.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f617c003a00 0x7f61981ee4e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f619815c8a0 0x7f61981f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 --2- 172.21.15.99:0/1550599382 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f619815c430 0x7f61981f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:42.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 >> 172.21.15.99:0/1550599382 conn(0x7f6198071060 msgr2=0x7f61980cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 shutdown_connections 2022-11-27T17:45:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:42.574+0000 7f619f297700 1 -- 172.21.15.99:0/1550599382 wait complete. 2022-11-27T17:45:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2022-11-27T17:45:42.588 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-11-27T17:45:42.588 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:42.589 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2022-11-27T17:45:42.604 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:42.603+0000 7f3491f81200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-11-27T17:45:42.615 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:42.615 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2022-11-27T17:45:42.670 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:42.669+0000 7f3491f81200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-11-27T17:45:42.680 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:42.681 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2022-11-27T17:45:42.746 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2022-11-27T17:45:42.747 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2022-11-27T17:45:42.747 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:42.746+0000 7f3491f81200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-11-27T17:45:42.773 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:42.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2022-11-27T17:45:42.802 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:42.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2022-11-27T17:45:42.830 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:42.827+0000 7f3491f81200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-11-27T17:45:42.870 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:42.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2022-11-27T17:45:42.937 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2022-11-27T17:45:42.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2022-11-27T17:45:42.964 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:42.962+0000 7f504b13c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-11-27T17:45:42.968 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:42.966+0000 7f3491f81200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-11-27T17:45:42.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new a82a87a1-4c37-42e5-b3c8-42f429d47bd7 0 2022-11-27T17:45:43.030 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.028+0000 7f504b13c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-11-27T17:45:43.107 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.106+0000 7f504b13c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-11-27T17:45:43.150 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.149+0000 7f8181421700 1 Processor -- start 2022-11-27T17:45:43.151 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.149+0000 7f8181421700 1 -- start start 2022-11-27T17:45:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.149+0000 7f8181421700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.149+0000 7f8181421700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 0x7f817c0c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.149+0000 7f8181421700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817c070890 con 0x7f817c0ce510 2022-11-27T17:45:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f8181421700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817c0c4b00 con 0x7f817c06ff80 2022-11-27T17:45:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f8181421700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c0cda30 con 0x7f817c0c4620 2022-11-27T17:45:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 0x7f817c0c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55462/0 (socket says 172.21.15.119:55462) 2022-11-27T17:45:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817affd700 1 -- 172.21.15.119:0/278646898 learned_addr learned my addr 172.21.15.119:0/278646898 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 -- 172.21.15.119:0/278646898 >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 msgr2=0x7f817c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 --2- 172.21.15.119:0/278646898 >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 -- 172.21.15.119:0/278646898 >> v1:172.21.15.119:6789/0 conn(0x7f817c0ce510 legacy=0x7f817c06c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 -- 172.21.15.119:0/278646898 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0cdd10 con 0x7f817c0c4620 2022-11-27T17:45:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817affd700 1 --2- 172.21.15.119:0/278646898 >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f817a7fc700 1 --2- 172.21.15.119:0/278646898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 0x7f817c0c3ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8170009840 tx=0x7f8170009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f928f358fa60b470 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f8179ffb700 1 -- 172.21.15.119:0/278646898 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170011830 con 0x7f817c0c4620 2022-11-27T17:45:43.160 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f8179ffb700 1 -- 172.21.15.119:0/278646898 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8170011990 con 0x7f817c0c4620 2022-11-27T17:45:43.160 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.150+0000 7f8179ffb700 1 -- 172.21.15.119:0/278646898 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170011d20 con 0x7f817c0c4620 2022-11-27T17:45:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 -- 172.21.15.119:0/278646898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 msgr2=0x7f817c0c3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 --2- 172.21.15.119:0/278646898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 0x7f817c0c3ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8170009840 tx=0x7f8170009b50 comp rx=0 tx=0).stop 2022-11-27T17:45:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 -- 172.21.15.119:0/278646898 shutdown_connections 2022-11-27T17:45:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 --2- 172.21.15.119:0/278646898 >> v2:172.21.15.119:3300/0 conn(0x7f817c06ff80 0x7f817c070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 --2- 172.21.15.119:0/278646898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c0c4620 0x7f817c0c3ca0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 -- 172.21.15.119:0/278646898 >> 172.21.15.119:0/278646898 conn(0x7f817c0b16c0 msgr2=0x7f817c0cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 -- 172.21.15.119:0/278646898 shutdown_connections 2022-11-27T17:45:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.151+0000 7f8181421700 1 -- 172.21.15.119:0/278646898 wait complete. 2022-11-27T17:45:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.153+0000 7f8181421700 1 Processor -- start 2022-11-27T17:45:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.153+0000 7f8181421700 1 -- start start 2022-11-27T17:45:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 0x7f817c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f817c0ce510 0x7f817c1449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c156b80 con 0x7f817c0c4620 2022-11-27T17:45:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817c1485c0 con 0x7f817c0ce510 2022-11-27T17:45:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f8181421700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817c147ea0 con 0x7f817c06ff80 2022-11-27T17:45:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f817affd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f817affd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50228/0 (socket says 172.21.15.119:50228) 2022-11-27T17:45:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 -- 172.21.15.119:0/1258174894 learned_addr learned my addr 172.21.15.119:0/1258174894 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.154+0000 7f817b7fe700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f817c0ce510 0x7f817c1449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817a7fc700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 0x7f817c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 -- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f817c0ce510 msgr2=0x7f817c1449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f817c0ce510 0x7f817c1449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 -- 172.21.15.119:0/1258174894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 msgr2=0x7f817c148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 0x7f817c148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 -- 172.21.15.119:0/1258174894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c009550 con 0x7f817c06ff80 2022-11-27T17:45:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817a7fc700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 0x7f817c148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:45:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f817affd700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f816c00ebe0 tx=0x7f816c00eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c00c850 con 0x7f817c06ff80 2022-11-27T17:45:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816c00c9b0 con 0x7f817c06ff80 2022-11-27T17:45:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81700094a0 con 0x7f817c06ff80 2022-11-27T17:45:43.168 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816c00cd20 con 0x7f817c06ff80 2022-11-27T17:45:43.168 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817c1564c0 con 0x7f817c06ff80 2022-11-27T17:45:43.168 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f816c019030 con 0x7f817c06ff80 2022-11-27T17:45:43.168 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.155+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f816c0066d0 con 0x7f817c06ff80 2022-11-27T17:45:43.168 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.156+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817c0cec60 con 0x7f817c06ff80 2022-11-27T17:45:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.158+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f816c021060 con 0x7f817c06ff80 2022-11-27T17:45:43.191 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.190+0000 7f504b13c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-11-27T17:45:43.244 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.242+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a82a87a1-4c37-42e5-b3c8-42f429d47bd7", "id": 0} v 0) v1 -- 0x7f817c073030 con 0x7f817c06ff80 2022-11-27T17:45:43.249 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.247+0000 7f81637fe700 1 -- 172.21.15.119:0/1258174894 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a82a87a1-4c37-42e5-b3c8-42f429d47bd7", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f816c017070 con 0x7f817c06ff80 2022-11-27T17:45:43.249 INFO:teuthology.orchestra.run.smithi119.stdout:0 2022-11-27T17:45:43.251 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 msgr2=0x7f817c0b2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.251 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f816c00ebe0 tx=0x7f816c00eef0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.251 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 shutdown_connections 2022-11-27T17:45:43.252 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f817c0ce510 0x7f817c1449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.252 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f817c06ff80 0x7f817c0b2a90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.252 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 --2- 172.21.15.119:0/1258174894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f817c0c4620 0x7f817c148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.252 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 >> 172.21.15.119:0/1258174894 conn(0x7f817c0b16c0 msgr2=0x7f817c0693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.253 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 shutdown_connections 2022-11-27T17:45:43.253 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.248+0000 7f8181421700 1 -- 172.21.15.119:0/1258174894 wait complete. 2022-11-27T17:45:43.262 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new a688c5d6-7c05-4bba-8590-c2029d25c345 1 2022-11-27T17:45:43.298 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:43.296+0000 7f3491f81200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-11-27T17:45:43.337 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.336+0000 7f504b13c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-11-27T17:45:43.371 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:43.369+0000 7f3491f81200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-11-27T17:45:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.381+0000 7f1916132700 1 Processor -- start 2022-11-27T17:45:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.381+0000 7f1916132700 1 -- start start 2022-11-27T17:45:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.381+0000 7f1916132700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f19100ccd20 0x7f19100b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.381+0000 7f1916132700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191015f390 con 0x7f191015efc0 2022-11-27T17:45:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.381+0000 7f1916132700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100ce050 con 0x7f19100b0590 2022-11-27T17:45:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f1916132700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19100b0a10 con 0x7f19100ccd20 2022-11-27T17:45:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f19100b0590 0x7f19100693f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:47498/0 (socket says 172.21.15.119:47498) 2022-11-27T17:45:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190ffff700 1 -- 172.21.15.119:0/3587693039 learned_addr learned my addr 172.21.15.119:0/3587693039 (peer_addr_for_me v1:172.21.15.119:0/0) 2022-11-27T17:45:43.392 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190effd700 1 --2- 172.21.15.119:0/3587693039 >> v2:172.21.15.99:3300/0 conn(0x7f19100ccd20 0x7f19100b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.392 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190e7fc700 1 -- 172.21.15.119:0/3587693039 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254431914 0 0) 0x7f19100ce050 con 0x7f19100b0590 2022-11-27T17:45:43.392 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190effd700 1 -- 172.21.15.119:0/3587693039 >> v1:172.21.15.119:6789/0 conn(0x7f19100b0590 legacy=0x7f19100693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190effd700 1 -- 172.21.15.119:0/3587693039 >> v1:172.21.15.99:6789/0 conn(0x7f191015efc0 legacy=0x7f19100cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190effd700 1 -- 172.21.15.119:0/3587693039 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100cc240 con 0x7f19100ccd20 2022-11-27T17:45:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190effd700 1 --2- 172.21.15.119:0/3587693039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f19100b0050 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009840 tx=0x7f18f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ce061202a9a562 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190e7fc700 1 -- 172.21.15.119:0/3587693039 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8011830 con 0x7f19100ccd20 2022-11-27T17:45:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190e7fc700 1 -- 172.21.15.119:0/3587693039 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f8011990 con 0x7f19100ccd20 2022-11-27T17:45:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.382+0000 7f190e7fc700 1 -- 172.21.15.119:0/3587693039 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8011d20 con 0x7f19100ccd20 2022-11-27T17:45:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 -- 172.21.15.119:0/3587693039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 msgr2=0x7f19100b0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 --2- 172.21.15.119:0/3587693039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f19100b0050 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009840 tx=0x7f18f8009b50 comp rx=0 tx=0).stop 2022-11-27T17:45:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 -- 172.21.15.119:0/3587693039 shutdown_connections 2022-11-27T17:45:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 --2- 172.21.15.119:0/3587693039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f19100b0050 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 -- 172.21.15.119:0/3587693039 >> 172.21.15.119:0/3587693039 conn(0x7f19100cb050 msgr2=0x7f19100cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 -- 172.21.15.119:0/3587693039 shutdown_connections 2022-11-27T17:45:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.383+0000 7f1916132700 1 -- 172.21.15.119:0/3587693039 wait complete. 2022-11-27T17:45:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.384+0000 7f1916132700 1 Processor -- start 2022-11-27T17:45:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.384+0000 7f1916132700 1 -- start start 2022-11-27T17:45:43.397 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 0x7f1910151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.397 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f191015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.397 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.400 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19100af3c0 con 0x7f19100ccd20 2022-11-27T17:45:43.400 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19100af5d0 con 0x7f191015efc0 2022-11-27T17:45:43.400 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f1916132700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19100af8b0 con 0x7f19100b0590 2022-11-27T17:45:43.400 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.401 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190f7fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 0x7f1910151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.401 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f191015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.401 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55482/0 (socket says 172.21.15.119:55482) 2022-11-27T17:45:43.401 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190ffff700 1 -- 172.21.15.119:0/3442865218 learned_addr learned my addr 172.21.15.119:0/3442865218 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.402 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190f7fe700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 0x7f1910151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50238/0 (socket says 172.21.15.119:50238) 2022-11-27T17:45:43.402 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 -- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 msgr2=0x7f1910151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.402 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 0x7f1910151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.402 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 -- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 msgr2=0x7f191015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.403 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.403 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190effd700 1 -- 172.21.15.119:0/3442865218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f80094a0 con 0x7f19100ccd20 2022-11-27T17:45:43.403 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.385+0000 7f190ffff700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.403 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f190effd700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f191015a0d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18f8016000 tx=0x7f18f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.404 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f801a7e0 con 0x7f19100ccd20 2022-11-27T17:45:43.404 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f801a940 con 0x7f19100ccd20 2022-11-27T17:45:43.404 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19100afb90 con 0x7f19100ccd20 2022-11-27T17:45:43.404 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f801acb0 con 0x7f19100ccd20 2022-11-27T17:45:43.405 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19100b0150 con 0x7f19100ccd20 2022-11-27T17:45:43.405 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f18f802b030 con 0x7f19100ccd20 2022-11-27T17:45:43.405 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.386+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f18f802ec60 con 0x7f19100ccd20 2022-11-27T17:45:43.408 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.387+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1910073030 con 0x7f19100ccd20 2022-11-27T17:45:43.409 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.389+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f18f8029070 con 0x7f19100ccd20 2022-11-27T17:45:43.474 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.472+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a688c5d6-7c05-4bba-8590-c2029d25c345", "id": 1} v 0) v1 -- 0x7f19100cc520 con 0x7f19100ccd20 2022-11-27T17:45:43.479 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.477+0000 7f1907fff700 1 -- 172.21.15.119:0/3442865218 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a688c5d6-7c05-4bba-8590-c2029d25c345", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f18f8015020 con 0x7f19100ccd20 2022-11-27T17:45:43.479 INFO:teuthology.orchestra.run.smithi119.stdout:1 2022-11-27T17:45:43.482 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 msgr2=0x7f191015a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.482 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f191015a0d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18f8016000 tx=0x7f18f8004750 comp rx=0 tx=0).stop 2022-11-27T17:45:43.482 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 shutdown_connections 2022-11-27T17:45:43.483 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f191015efc0 0x7f191015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.483 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f19100b0590 0x7f1910151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.483 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 --2- 172.21.15.119:0/3442865218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19100ccd20 0x7f191015a0d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.483 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 >> 172.21.15.119:0/3442865218 conn(0x7f19100cb050 msgr2=0x7f19100ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.484 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 shutdown_connections 2022-11-27T17:45:43.484 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.478+0000 7f1916132700 1 -- 172.21.15.119:0/3442865218 wait complete. 2022-11-27T17:45:43.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 04ae94e5-c567-40e0-b979-63a599c1f2e6 2 2022-11-27T17:45:43.499 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:43.498+0000 7f3491f81200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-11-27T17:45:43.592 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:43.590+0000 7f3491f81200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-11-27T17:45:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 Processor -- start 2022-11-27T17:45:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 -- start start 2022-11-27T17:45:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540693f0 con 0x7f645406d920 2022-11-27T17:45:43.623 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645406dda0 con 0x7f64540cf530 2022-11-27T17:45:43.623 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.610+0000 7f645a063700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645406a1c0 con 0x7f64540ce7b0 2022-11-27T17:45:43.624 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f6452ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.624 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.624 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f6452ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55484/0 (socket says 172.21.15.119:55484) 2022-11-27T17:45:43.624 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50244/0 (socket says 172.21.15.119:50244) 2022-11-27T17:45:43.625 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f6452ffd700 1 -- 172.21.15.119:0/3928207496 learned_addr learned my addr 172.21.15.119:0/3928207496 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.625 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 -- 172.21.15.119:0/3928207496 >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 msgr2=0x7f645406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:43.625 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 --2- 172.21.15.119:0/3928207496 >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.626 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 -- 172.21.15.119:0/3928207496 >> v1:172.21.15.119:6789/0 conn(0x7f645406d920 legacy=0x7f645415d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.626 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 -- 172.21.15.119:0/3928207496 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454069640 con 0x7f64540ce7b0 2022-11-27T17:45:43.626 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64527fc700 1 -- 172.21.15.119:0/3928207496 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032560934 0 0) 0x7f64540693f0 con 0x7f645406d920 2022-11-27T17:45:43.627 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f6452ffd700 1 --2- 172.21.15.119:0/3928207496 >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.627 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64537fe700 1 --2- 172.21.15.119:0/3928207496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f644400dc60 tx=0x7f644400df70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e26a9ea502040514 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.627 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64527fc700 1 -- 172.21.15.119:0/3928207496 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444012070 con 0x7f64540ce7b0 2022-11-27T17:45:43.628 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64527fc700 1 -- 172.21.15.119:0/3928207496 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644400d2e0 con 0x7f64540ce7b0 2022-11-27T17:45:43.628 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.611+0000 7f64527fc700 1 -- 172.21.15.119:0/3928207496 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644400f900 con 0x7f64540ce7b0 2022-11-27T17:45:43.628 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 -- 172.21.15.119:0/3928207496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 msgr2=0x7f6454159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.629 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 --2- 172.21.15.119:0/3928207496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f644400dc60 tx=0x7f644400df70 comp rx=0 tx=0).stop 2022-11-27T17:45:43.629 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 -- 172.21.15.119:0/3928207496 shutdown_connections 2022-11-27T17:45:43.629 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 --2- 172.21.15.119:0/3928207496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f64540ce7b0 0x7f6454159e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.630 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 --2- 172.21.15.119:0/3928207496 >> v2:172.21.15.119:3300/0 conn(0x7f64540cf530 0x7f645406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.630 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 -- 172.21.15.119:0/3928207496 >> 172.21.15.119:0/3928207496 conn(0x7f64540b16c0 msgr2=0x7f64540cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.631 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 -- 172.21.15.119:0/3928207496 shutdown_connections 2022-11-27T17:45:43.631 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.612+0000 7f645a063700 1 -- 172.21.15.119:0/3928207496 wait complete. 2022-11-27T17:45:43.631 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.613+0000 7f645a063700 1 Processor -- start 2022-11-27T17:45:43.631 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.613+0000 7f645a063700 1 -- start start 2022-11-27T17:45:43.632 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.632 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 0x7f64540b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.632 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 0x7f64540b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.632 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64541525c0 con 0x7f64540ce7b0 2022-11-27T17:45:43.632 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64541528a0 con 0x7f64540cf530 2022-11-27T17:45:43.633 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f645a063700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6454152b80 con 0x7f645406d920 2022-11-27T17:45:43.633 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f64537fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.633 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f6453fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 0x7f64540b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.633 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f64537fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50250/0 (socket says 172.21.15.119:50250) 2022-11-27T17:45:43.634 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f64537fe700 1 -- 172.21.15.119:0/1377832841 learned_addr learned my addr 172.21.15.119:0/1377832841 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.634 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f6452ffd700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 0x7f64540b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.634 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f6453fff700 1 -- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 msgr2=0x7f64541ecc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.634 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.614+0000 7f6453fff700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.634 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f6453fff700 1 -- 172.21.15.119:0/1377832841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 msgr2=0x7f64540b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.635 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f6453fff700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 0x7f64540b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.635 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f6453fff700 1 -- 172.21.15.119:0/1377832841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448009550 con 0x7f64540cf530 2022-11-27T17:45:43.635 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f6452ffd700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 0x7f64540b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:45:43.635 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f64537fe700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.635 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f6453fff700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 0x7f64540b36f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c7c0 tx=0x7f644c00cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.636 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c003e30 con 0x7f64540cf530 2022-11-27T17:45:43.636 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c005070 con 0x7f64540cf530 2022-11-27T17:45:43.636 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644400d940 con 0x7f64540cf530 2022-11-27T17:45:43.636 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c005420 con 0x7f64540cf530 2022-11-27T17:45:43.636 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64541ed190 con 0x7f64540cf530 2022-11-27T17:45:43.637 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f644c019030 con 0x7f64540cf530 2022-11-27T17:45:43.637 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.615+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f644c01c440 con 0x7f64540cf530 2022-11-27T17:45:43.637 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.616+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6454065ac0 con 0x7f64540cf530 2022-11-27T17:45:43.637 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.618+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f644c017070 con 0x7f64540cf530 2022-11-27T17:45:43.684 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.682+0000 7f504b13c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-11-27T17:45:43.702 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.701+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "04ae94e5-c567-40e0-b979-63a599c1f2e6", "id": 2} v 0) v1 -- 0x7f6454069920 con 0x7f64540cf530 2022-11-27T17:45:43.710 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.708+0000 7f643bfff700 1 -- 172.21.15.119:0/1377832841 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04ae94e5-c567-40e0-b979-63a599c1f2e6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f644c005580 con 0x7f64540cf530 2022-11-27T17:45:43.711 INFO:teuthology.orchestra.run.smithi119.stdout:2 2022-11-27T17:45:43.712 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 msgr2=0x7f64540b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.712 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 0x7f64540b36f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c7c0 tx=0x7f644c00cad0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.713 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 shutdown_connections 2022-11-27T17:45:43.713 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64540cf530 0x7f64540b36f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.713 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f645406d920 0x7f64541ecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.713 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 --2- 172.21.15.119:0/1377832841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64540ce7b0 0x7f64540b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.714 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.709+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 >> 172.21.15.119:0/1377832841 conn(0x7f64540b16c0 msgr2=0x7f64541506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.714 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.710+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 shutdown_connections 2022-11-27T17:45:43.714 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.710+0000 7f645a063700 1 -- 172.21.15.119:0/1377832841 wait complete. 2022-11-27T17:45:43.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e 3 2022-11-27T17:45:43.762 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.760+0000 7f504b13c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-11-27T17:45:43.846 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.844+0000 7fc4ffc23700 1 Processor -- start 2022-11-27T17:45:43.846 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.844+0000 7fc4ffc23700 1 -- start start 2022-11-27T17:45:43.854 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4ffc23700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 0x7fc4f80b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.854 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4ffc23700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 0x7fc4f806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.854 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4ffc23700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f80b1ca0 con 0x7fc4f81598c0 2022-11-27T17:45:43.855 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4ffc23700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f80cc830 con 0x7fc4f806d120 2022-11-27T17:45:43.855 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4ffc23700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80b3770 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.855 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd9bf700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc4f81598c0 0x7fc4f80d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:47518/0 (socket says 172.21.15.119:47518) 2022-11-27T17:45:43.855 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd9bf700 1 -- 172.21.15.119:0/1068538098 learned_addr learned my addr 172.21.15.119:0/1068538098 (peer_addr_for_me v1:172.21.15.119:0/0) 2022-11-27T17:45:43.856 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fe1c0700 1 --2- 172.21.15.119:0/1068538098 >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 0x7fc4f806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.856 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd1be700 1 --2- 172.21.15.119:0/1068538098 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 0x7fc4f80b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.856 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd1be700 1 -- 172.21.15.119:0/1068538098 >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 msgr2=0x7fc4f806d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:43.856 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd1be700 1 --2- 172.21.15.119:0/1068538098 >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 0x7fc4f806d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.857 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd1be700 1 -- 172.21.15.119:0/1068538098 >> v1:172.21.15.119:6789/0 conn(0x7fc4f81598c0 legacy=0x7fc4f80d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.857 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fd1be700 1 -- 172.21.15.119:0/1068538098 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f80d0cb0 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.857 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.845+0000 7fc4fe1c0700 1 --2- 172.21.15.119:0/1068538098 >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 0x7fc4f806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.857 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4fd1be700 1 --2- 172.21.15.119:0/1068538098 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 0x7fc4f80b3230 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009840 tx=0x7fc4e8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92acad67c2a8ba0f server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.858 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4fc9bd700 1 -- 172.21.15.119:0/1068538098 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8011830 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.858 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4fc9bd700 1 -- 172.21.15.119:0/1068538098 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e8011990 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.858 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4fc9bd700 1 -- 172.21.15.119:0/1068538098 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8011d20 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.858 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 -- 172.21.15.119:0/1068538098 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 msgr2=0x7fc4f80b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.859 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/1068538098 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 0x7fc4f80b3230 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009840 tx=0x7fc4e8009b50 comp rx=0 tx=0).stop 2022-11-27T17:45:43.859 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 -- 172.21.15.119:0/1068538098 shutdown_connections 2022-11-27T17:45:43.859 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/1068538098 >> v2:172.21.15.119:3300/0 conn(0x7fc4f806d120 0x7fc4f806d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.860 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/1068538098 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f80b2dc0 0x7fc4f80b3230 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.860 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.846+0000 7fc4ffc23700 1 -- 172.21.15.119:0/1068538098 >> 172.21.15.119:0/1068538098 conn(0x7fc4f8071060 msgr2=0x7fc4f8071430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.860 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.847+0000 7fc4ffc23700 1 -- 172.21.15.119:0/1068538098 shutdown_connections 2022-11-27T17:45:43.860 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.847+0000 7fc4ffc23700 1 -- 172.21.15.119:0/1068538098 wait complete. 2022-11-27T17:45:43.861 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 Processor -- start 2022-11-27T17:45:43.861 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 -- start start 2022-11-27T17:45:43.861 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.861 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 0x7fc4f80c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.862 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:43.862 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f815ad80 con 0x7fc4f80b2dc0 2022-11-27T17:45:43.862 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80c7ac0 con 0x7fc4f81598c0 2022-11-27T17:45:43.862 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4ffc23700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f80c7da0 con 0x7fc4f806d120 2022-11-27T17:45:43.863 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.848+0000 7fc4fd9bf700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.863 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.863 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55506/0 (socket says 172.21.15.119:55506) 2022-11-27T17:45:43.863 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fd9bf700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50262/0 (socket says 172.21.15.119:50262) 2022-11-27T17:45:43.864 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 -- 172.21.15.119:0/560288737 learned_addr learned my addr 172.21.15.119:0/560288737 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:43.864 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fd1be700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 0x7fc4f80c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:43.864 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 -- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 msgr2=0x7fc4f80c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.864 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.865 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 -- 172.21.15.119:0/560288737 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 msgr2=0x7fc4f80c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.865 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 0x7fc4f80c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.865 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 -- 172.21.15.119:0/560288737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f0009a70 con 0x7fc4f81598c0 2022-11-27T17:45:43.865 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fd9bf700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:43.865 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fd1be700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 0x7fc4f80c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:45:43.866 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4fe1c0700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000efc0 tx=0x7fc4f000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:43.866 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f000cb40 con 0x7fc4f81598c0 2022-11-27T17:45:43.866 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f000f040 con 0x7fc4f81598c0 2022-11-27T17:45:43.866 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e80094a0 con 0x7fc4f81598c0 2022-11-27T17:45:43.867 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0003a20 con 0x7fc4f81598c0 2022-11-27T17:45:43.867 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f815aa10 con 0x7fc4f81598c0 2022-11-27T17:45:43.867 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc4f0014480 con 0x7fc4f81598c0 2022-11-27T17:45:43.867 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.849+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fc4f001d990 con 0x7fc4f81598c0 2022-11-27T17:45:43.867 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.850+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f815aef0 con 0x7fc4f81598c0 2022-11-27T17:45:43.868 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.852+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7fc4f0018030 con 0x7fc4f81598c0 2022-11-27T17:45:43.893 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.891+0000 7f504b13c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-11-27T17:45:43.936 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.934+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e", "id": 3} v 0) v1 -- 0x7fc4f8073030 con 0x7fc4f81598c0 2022-11-27T17:45:43.943 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.941+0000 7fc4e67fc700 1 -- 172.21.15.119:0/560288737 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc4f001de40 con 0x7fc4f81598c0 2022-11-27T17:45:43.944 INFO:teuthology.orchestra.run.smithi119.stdout:3 2022-11-27T17:45:43.946 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 msgr2=0x7fc4f80c5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:43.946 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc4f000efc0 tx=0x7fc4f000c3a0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.947 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 shutdown_connections 2022-11-27T17:45:43.947 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f81598c0 0x7fc4f80c5a20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f806d120 0x7fc4f80c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 --2- 172.21.15.119:0/560288737 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4f80b2dc0 0x7fc4f80c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 >> 172.21.15.119:0/560288737 conn(0x7fc4f8071060 msgr2=0x7fc4f80b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 shutdown_connections 2022-11-27T17:45:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:43.943+0000 7fc4ffc23700 1 -- 172.21.15.119:0/560288737 wait complete. 2022-11-27T17:45:43.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new f36aa763-c864-431b-a1f8-f7a5d0253246 4 2022-11-27T17:45:43.976 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:43.974+0000 7f504b13c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-11-27T17:45:44.012 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:44.010+0000 7f3491f81200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-11-27T17:45:44.096 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.094+0000 7f9302800700 1 Processor -- start 2022-11-27T17:45:44.096 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.094+0000 7f9302800700 1 -- start start 2022-11-27T17:45:44.099 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9302800700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.099 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9302800700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f92fc0b0760 0x7f92fc0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9302800700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92fc0ce050 con 0x7f92fc0cd600 2022-11-27T17:45:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9302800700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92fc0cc240 con 0x7f92fc0b0760 2022-11-27T17:45:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9302800700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0cc520 con 0x7f92fc15ba40 2022-11-27T17:45:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f9300d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f92fc0b0760 0x7f92fc0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50268/0 (socket says 172.21.15.119:50268) 2022-11-27T17:45:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 -- 172.21.15.119:0/346985636 learned_addr learned my addr 172.21.15.119:0/346985636 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 -- 172.21.15.119:0/346985636 >> v2:172.21.15.119:3300/0 conn(0x7f92fc0b0760 msgr2=0x7f92fc0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 --2- 172.21.15.119:0/346985636 >> v2:172.21.15.119:3300/0 conn(0x7f92fc0b0760 0x7f92fc0cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 -- 172.21.15.119:0/346985636 >> v1:172.21.15.99:6789/0 conn(0x7f92fc0cd600 legacy=0x7f92fc0b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 -- 172.21.15.119:0/346985636 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc0cc800 con 0x7f92fc15ba40 2022-11-27T17:45:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92fbfff700 1 --2- 172.21.15.119:0/346985636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0098f0 tx=0x7f92ec009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f694f5cf836afe server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.095+0000 7f92faffd700 1 -- 172.21.15.119:0/346985636 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec011830 con 0x7f92fc15ba40 2022-11-27T17:45:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f92faffd700 1 -- 172.21.15.119:0/346985636 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec011990 con 0x7f92fc15ba40 2022-11-27T17:45:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f92faffd700 1 -- 172.21.15.119:0/346985636 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec011d20 con 0x7f92fc15ba40 2022-11-27T17:45:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 -- 172.21.15.119:0/346985636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 msgr2=0x7f92fc0ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 --2- 172.21.15.119:0/346985636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0098f0 tx=0x7f92ec009c00 comp rx=0 tx=0).stop 2022-11-27T17:45:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 -- 172.21.15.119:0/346985636 shutdown_connections 2022-11-27T17:45:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 --2- 172.21.15.119:0/346985636 >> v2:172.21.15.119:3300/0 conn(0x7f92fc0b0760 0x7f92fc0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 --2- 172.21.15.119:0/346985636 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0ccd20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 -- 172.21.15.119:0/346985636 >> 172.21.15.119:0/346985636 conn(0x7f92fc0cb050 msgr2=0x7f92fc0cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 -- 172.21.15.119:0/346985636 shutdown_connections 2022-11-27T17:45:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.096+0000 7f9302800700 1 -- 172.21.15.119:0/346985636 wait complete. 2022-11-27T17:45:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.098+0000 7f9302800700 1 Processor -- start 2022-11-27T17:45:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.098+0000 7f9302800700 1 -- start start 2022-11-27T17:45:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92fc0cd600 0x7f92fc158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc1f5730 con 0x7f92fc0cd600 2022-11-27T17:45:44.111 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc1590c0 con 0x7f92fc0b0760 2022-11-27T17:45:44.111 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9302800700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0c6d60 con 0x7f92fc15ba40 2022-11-27T17:45:44.111 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9300d9d700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.111 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f92fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.112 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9300d9d700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50276/0 (socket says 172.21.15.119:50276) 2022-11-27T17:45:44.112 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f92fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55512/0 (socket says 172.21.15.119:55512) 2022-11-27T17:45:44.112 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f9300d9d700 1 -- 172.21.15.119:0/3240127433 learned_addr learned my addr 172.21.15.119:0/3240127433 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f92fb7fe700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92fc0cd600 0x7f92fc158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f92fbfff700 1 -- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 msgr2=0x7f92fc0c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.099+0000 7f92fbfff700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92fbfff700 1 -- 172.21.15.119:0/3240127433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92fc0cd600 msgr2=0x7f92fc158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92fbfff700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92fc0cd600 0x7f92fc158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92fbfff700 1 -- 172.21.15.119:0/3240127433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009550 con 0x7f92fc0b0760 2022-11-27T17:45:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92fbfff700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f92ec016000 tx=0x7f92ec004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec004400 con 0x7f92fc0b0760 2022-11-27T17:45:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec01f070 con 0x7f92fc0b0760 2022-11-27T17:45:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0038e0 con 0x7f92fc0b0760 2022-11-27T17:45:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc1f4bb0 con 0x7f92fc0b0760 2022-11-27T17:45:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc1f5170 con 0x7f92fc0b0760 2022-11-27T17:45:44.116 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f92ec02a020 con 0x7f92fc0b0760 2022-11-27T17:45:44.116 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.100+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f92ec01aa90 con 0x7f92fc0b0760 2022-11-27T17:45:44.116 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.101+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc073030 con 0x7f92fc0b0760 2022-11-27T17:45:44.116 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.103+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f92ec015020 con 0x7f92fc0b0760 2022-11-27T17:45:44.187 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.185+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f36aa763-c864-431b-a1f8-f7a5d0253246", "id": 4} v 0) v1 -- 0x7f92fc0ce330 con 0x7f92fc0b0760 2022-11-27T17:45:44.192 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.191+0000 7f92f8ff9700 1 -- 172.21.15.119:0/3240127433 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f36aa763-c864-431b-a1f8-f7a5d0253246", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f92ec01aca0 con 0x7f92fc0b0760 2022-11-27T17:45:44.193 INFO:teuthology.orchestra.run.smithi119.stdout:4 2022-11-27T17:45:44.194 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 msgr2=0x7f92fc15ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.194 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f92ec016000 tx=0x7f92ec004750 comp rx=0 tx=0).stop 2022-11-27T17:45:44.195 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 shutdown_connections 2022-11-27T17:45:44.195 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f92fc15ba40 0x7f92fc0c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.195 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92fc0b0760 0x7f92fc15ee90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.195 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 --2- 172.21.15.119:0/3240127433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92fc0cd600 0x7f92fc158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.195 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 >> 172.21.15.119:0/3240127433 conn(0x7f92fc0cb050 msgr2=0x7f92fc0ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.196 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 shutdown_connections 2022-11-27T17:45:44.196 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.192+0000 7f9302800700 1 -- 172.21.15.119:0/3240127433 wait complete. 2022-11-27T17:45:44.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new cafb58cd-ca0e-4dae-bf69-9ac81df4074e 5 2022-11-27T17:45:44.325 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 Processor -- start 2022-11-27T17:45:44.326 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 -- start start 2022-11-27T17:45:44.332 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f937015bb10 0x7f93700693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.332 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f937015bee0 0x7f937006c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.332 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937006c050 con 0x7f937006cd70 2022-11-27T17:45:44.332 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9370069c40 con 0x7f937015bee0 2022-11-27T17:45:44.333 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f93758bd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9370069f20 con 0x7f937015bb10 2022-11-27T17:45:44.333 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.324+0000 7f936effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f937015bb10 0x7f93700693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.333 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f937015bb10 0x7f93700693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55518/0 (socket says 172.21.15.119:55518) 2022-11-27T17:45:44.333 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 -- 172.21.15.119:0/3779688364 learned_addr learned my addr 172.21.15.119:0/3779688364 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.333 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936e7fc700 1 --2- 172.21.15.119:0/3779688364 >> v2:172.21.15.99:3300/0 conn(0x7f937015bee0 0x7f937006c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.334 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 -- 172.21.15.119:0/3779688364 >> v2:172.21.15.99:3300/0 conn(0x7f937015bee0 msgr2=0x7f937006c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:44.334 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 --2- 172.21.15.119:0/3779688364 >> v2:172.21.15.99:3300/0 conn(0x7f937015bee0 0x7f937006c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.334 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 -- 172.21.15.119:0/3779688364 >> v1:172.21.15.99:6789/0 conn(0x7f937006cd70 legacy=0x7f937006d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.334 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 -- 172.21.15.119:0/3779688364 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937006a200 con 0x7f937015bb10 2022-11-27T17:45:44.335 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936effd700 1 --2- 172.21.15.119:0/3779688364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f93700693f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9370072a00 tx=0x7f9360009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d86289e2f934671 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.335 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936dffb700 1 -- 172.21.15.119:0/3779688364 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000b870 con 0x7f937015bb10 2022-11-27T17:45:44.335 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936dffb700 1 -- 172.21.15.119:0/3779688364 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936000b9d0 con 0x7f937015bb10 2022-11-27T17:45:44.335 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.325+0000 7f936dffb700 1 -- 172.21.15.119:0/3779688364 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000bd60 con 0x7f937015bb10 2022-11-27T17:45:44.335 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 -- 172.21.15.119:0/3779688364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 msgr2=0x7f93700693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.336 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 --2- 172.21.15.119:0/3779688364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f93700693f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9370072a00 tx=0x7f9360009bd0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.336 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 -- 172.21.15.119:0/3779688364 shutdown_connections 2022-11-27T17:45:44.336 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 --2- 172.21.15.119:0/3779688364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f93700693f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.336 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 --2- 172.21.15.119:0/3779688364 >> v2:172.21.15.99:3300/0 conn(0x7f937015bee0 0x7f937006c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 -- 172.21.15.119:0/3779688364 >> 172.21.15.119:0/3779688364 conn(0x7f93700b16c0 msgr2=0x7f93700cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 -- 172.21.15.119:0/3779688364 shutdown_connections 2022-11-27T17:45:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.326+0000 7f93758bd700 1 -- 172.21.15.119:0/3779688364 wait complete. 2022-11-27T17:45:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.327+0000 7f93758bd700 1 Processor -- start 2022-11-27T17:45:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.327+0000 7f93758bd700 1 -- start start 2022-11-27T17:45:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f9370143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f937015bee0 0x7f9370145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370144500 con 0x7f937015bee0 2022-11-27T17:45:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93701447e0 con 0x7f937015bb10 2022-11-27T17:45:44.339 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f93758bd700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9370144ac0 con 0x7f937006cd70 2022-11-27T17:45:44.339 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936effd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.339 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f9370143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.339 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936effd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50284/0 (socket says 172.21.15.119:50284) 2022-11-27T17:45:44.340 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936effd700 1 -- 172.21.15.119:0/3887422288 learned_addr learned my addr 172.21.15.119:0/3887422288 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.340 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936f7fe700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f937015bee0 0x7f9370145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.340 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 -- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 msgr2=0x7f93701e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.340 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.340 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 -- 172.21.15.119:0/3887422288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f937015bee0 msgr2=0x7f9370145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f937015bee0 0x7f9370145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 -- 172.21.15.119:0/3887422288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360009550 con 0x7f937015bb10 2022-11-27T17:45:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936effd700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f936e7fc700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f9370143ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f937006ff30 tx=0x7f936400d880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.328+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364017070 con 0x7f937015bb10 2022-11-27T17:45:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364004020 con 0x7f937015bb10 2022-11-27T17:45:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9370144da0 con 0x7f937015bb10 2022-11-27T17:45:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640041d0 con 0x7f937015bb10 2022-11-27T17:45:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93701e4c00 con 0x7f937015bb10 2022-11-27T17:45:44.343 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f9364004330 con 0x7f937015bb10 2022-11-27T17:45:44.343 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.329+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f9364012b60 con 0x7f937015bb10 2022-11-27T17:45:44.343 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.330+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9370065ac0 con 0x7f937015bb10 2022-11-27T17:45:44.344 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.332+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f9364012d70 con 0x7f937015bb10 2022-11-27T17:45:44.428 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.427+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cafb58cd-ca0e-4dae-bf69-9ac81df4074e", "id": 5} v 0) v1 -- 0x7f937006c310 con 0x7f937015bb10 2022-11-27T17:45:44.429 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:44.428+0000 7f504b13c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-11-27T17:45:44.438 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.436+0000 7f9357fff700 1 -- 172.21.15.119:0/3887422288 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cafb58cd-ca0e-4dae-bf69-9ac81df4074e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f93640125f0 con 0x7f937015bb10 2022-11-27T17:45:44.439 INFO:teuthology.orchestra.run.smithi119.stdout:5 2022-11-27T17:45:44.441 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 msgr2=0x7f9370143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.442 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f9370143ab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f937006ff30 tx=0x7f936400d880 comp rx=0 tx=0).stop 2022-11-27T17:45:44.442 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 shutdown_connections 2022-11-27T17:45:44.442 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f937015bee0 0x7f9370145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.443 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f937006cd70 0x7f93701e6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.443 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 --2- 172.21.15.119:0/3887422288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f937015bb10 0x7f9370143ab0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.444 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 >> 172.21.15.119:0/3887422288 conn(0x7f93700b16c0 msgr2=0x7f93700ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.444 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 shutdown_connections 2022-11-27T17:45:44.445 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.437+0000 7f93758bd700 1 -- 172.21.15.119:0/3887422288 wait complete. 2022-11-27T17:45:44.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 46754ecc-48dc-417a-be57-c27f20466fe1 6 2022-11-27T17:45:44.567 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:44.565+0000 7f3491f81200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-11-27T17:45:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.571+0000 7fed6ce9e700 1 Processor -- start 2022-11-27T17:45:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.571+0000 7fed6ce9e700 1 -- start start 2022-11-27T17:45:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.572+0000 7fed6ce9e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed6806daf0 0x7fed6815d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.572+0000 7fed6ce9e700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed680693f0 con 0x7fed680cf530 2022-11-27T17:45:44.577 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.572+0000 7fed6ce9e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6806a1c0 con 0x7fed680ce7b0 2022-11-27T17:45:44.577 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.572+0000 7fed6ce9e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed68069640 con 0x7fed6806daf0 2022-11-27T17:45:44.577 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fed680ce7b0 0x7fed68159e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:47552/0 (socket says 172.21.15.119:47552) 2022-11-27T17:45:44.577 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6659c700 1 -- 172.21.15.119:0/948503297 learned_addr learned my addr 172.21.15.119:0/948503297 (peer_addr_for_me v1:172.21.15.119:0/0) 2022-11-27T17:45:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed66d9d700 1 --2- 172.21.15.119:0/948503297 >> v2:172.21.15.119:3300/0 conn(0x7fed6806daf0 0x7fed6815d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728544815 0 0) 0x7fed6806a1c0 con 0x7fed680ce7b0 2022-11-27T17:45:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c0035b0 con 0x7fed680ce7b0 2022-11-27T17:45:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed66d9d700 1 -- 172.21.15.119:0/948503297 >> v1:172.21.15.119:6789/0 conn(0x7fed680ce7b0 legacy=0x7fed68159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed66d9d700 1 -- 172.21.15.119:0/948503297 >> v1:172.21.15.99:6789/0 conn(0x7fed680cf530 legacy=0x7fed6806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765810724 0 0) 0x7fed680693f0 con 0x7fed680cf530 2022-11-27T17:45:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed66d9d700 1 -- 172.21.15.119:0/948503297 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68069920 con 0x7fed6806daf0 2022-11-27T17:45:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed66d9d700 1 --2- 172.21.15.119:0/948503297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed6806daf0 0x7fed6815d7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed60009d90 tx=0x7fed6000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e3614aa684ebb0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6000e070 con 0x7fed6806daf0 2022-11-27T17:45:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6000bb20 con 0x7fed6806daf0 2022-11-27T17:45:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.573+0000 7fed6559a700 1 -- 172.21.15.119:0/948503297 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6000bed0 con 0x7fed6806daf0 2022-11-27T17:45:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 -- 172.21.15.119:0/948503297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed6806daf0 msgr2=0x7fed6815d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/948503297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed6806daf0 0x7fed6815d7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed60009d90 tx=0x7fed6000b330 comp rx=0 tx=0).stop 2022-11-27T17:45:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 -- 172.21.15.119:0/948503297 shutdown_connections 2022-11-27T17:45:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/948503297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed6806daf0 0x7fed6815d7f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 -- 172.21.15.119:0/948503297 >> 172.21.15.119:0/948503297 conn(0x7fed680b16c0 msgr2=0x7fed680cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.582 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 -- 172.21.15.119:0/948503297 shutdown_connections 2022-11-27T17:45:44.582 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.574+0000 7fed6ce9e700 1 -- 172.21.15.119:0/948503297 wait complete. 2022-11-27T17:45:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.575+0000 7fed6ce9e700 1 Processor -- start 2022-11-27T17:45:44.586 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.575+0000 7fed6ce9e700 1 -- start start 2022-11-27T17:45:44.586 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.586 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed680ce7b0 0x7fed680cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.587 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.587 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed68150bc0 con 0x7fed680ce7b0 2022-11-27T17:45:44.587 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed68150ea0 con 0x7fed680cf530 2022-11-27T17:45:44.588 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6ce9e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed68151180 con 0x7fed6806daf0 2022-11-27T17:45:44.588 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6659c700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.589 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.589 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6659c700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50294/0 (socket says 172.21.15.119:50294) 2022-11-27T17:45:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55538/0 (socket says 172.21.15.119:55538) 2022-11-27T17:45:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6659c700 1 -- 172.21.15.119:0/1094931278 learned_addr learned my addr 172.21.15.119:0/1094931278 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.591 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed65d9b700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed680ce7b0 0x7fed680cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.591 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 -- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 msgr2=0x7fed68150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.592 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.592 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 -- 172.21.15.119:0/1094931278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed680ce7b0 msgr2=0x7fed680cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.592 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed680ce7b0 0x7fed680cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed66d9d700 1 -- 172.21.15.119:0/1094931278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60009a70 con 0x7fed680cf530 2022-11-27T17:45:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.576+0000 7fed6659c700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed66d9d700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed60016000 tx=0x7fed60004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6001b810 con 0x7fed680cf530 2022-11-27T17:45:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed60020070 con 0x7fed680cf530 2022-11-27T17:45:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6001bbd0 con 0x7fed680cf530 2022-11-27T17:45:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6814b570 con 0x7fed680cf530 2022-11-27T17:45:44.596 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed68151ca0 con 0x7fed680cf530 2022-11-27T17:45:44.596 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fed6002e070 con 0x7fed680cf530 2022-11-27T17:45:44.596 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.577+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fed60022070 con 0x7fed680cf530 2022-11-27T17:45:44.597 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.578+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed68065ac0 con 0x7fed680cf530 2022-11-27T17:45:44.597 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.581+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7fed6002c060 con 0x7fed680cf530 2022-11-27T17:45:44.665 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.663+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "46754ecc-48dc-417a-be57-c27f20466fe1", "id": 6} v 0) v1 -- 0x7fed6814ba30 con 0x7fed680cf530 2022-11-27T17:45:44.673 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed56ffd700 1 -- 172.21.15.119:0/1094931278 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "46754ecc-48dc-417a-be57-c27f20466fe1", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fed6001be00 con 0x7fed680cf530 2022-11-27T17:45:44.673 INFO:teuthology.orchestra.run.smithi119.stdout:6 2022-11-27T17:45:44.675 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 msgr2=0x7fed680cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.676 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed60016000 tx=0x7fed60004ad0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.676 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 shutdown_connections 2022-11-27T17:45:44.676 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed680cf530 0x7fed680cb040 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.677 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.671+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed6806daf0 0x7fed68150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.677 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.672+0000 7fed6ce9e700 1 --2- 172.21.15.119:0/1094931278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed680ce7b0 0x7fed680cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.678 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.672+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 >> 172.21.15.119:0/1094931278 conn(0x7fed680b16c0 msgr2=0x7fed680b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.678 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.672+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 shutdown_connections 2022-11-27T17:45:44.678 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.672+0000 7fed6ce9e700 1 -- 172.21.15.119:0/1094931278 wait complete. 2022-11-27T17:45:44.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 48b889c7-f370-4749-bb57-a26ebf4eec3c 7 2022-11-27T17:45:44.720 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:44.718+0000 7f3491f81200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-11-27T17:45:44.808 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.806+0000 7f83f7c8a700 1 Processor -- start 2022-11-27T17:45:44.809 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 -- start start 2022-11-27T17:45:44.811 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f83f015bee0 0x7f83f006c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.811 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f006cd70 0x7f83f006d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.811 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f006c050 con 0x7f83f015bb10 2022-11-27T17:45:44.812 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f0069c40 con 0x7f83f015bee0 2022-11-27T17:45:44.812 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.807+0000 7f83f7c8a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f0069f20 con 0x7f83f006cd70 2022-11-27T17:45:44.812 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f006cd70 0x7f83f006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.812 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f006cd70 0x7f83f006d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55544/0 (socket says 172.21.15.119:55544) 2022-11-27T17:45:44.813 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 -- 172.21.15.119:0/301480464 learned_addr learned my addr 172.21.15.119:0/301480464 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.813 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f5225700 1 --2- 172.21.15.119:0/301480464 >> v2:172.21.15.99:3300/0 conn(0x7f83f015bee0 0x7f83f006c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.813 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 -- 172.21.15.119:0/301480464 >> v2:172.21.15.99:3300/0 conn(0x7f83f015bee0 msgr2=0x7f83f006c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:44.813 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 --2- 172.21.15.119:0/301480464 >> v2:172.21.15.99:3300/0 conn(0x7f83f015bee0 0x7f83f006c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.813 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 -- 172.21.15.119:0/301480464 >> v1:172.21.15.99:6789/0 conn(0x7f83f015bb10 legacy=0x7f83f00693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.814 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 -- 172.21.15.119:0/301480464 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f006a200 con 0x7f83f006cd70 2022-11-27T17:45:44.814 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f6227700 1 --2- 172.21.15.119:0/301480464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f006cd70 0x7f83f006d390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83f006b430 tx=0x7f83e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e4d26fc3d41e1d1 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.814 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f4a24700 1 -- 172.21.15.119:0/301480464 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800e070 con 0x7f83f006cd70 2022-11-27T17:45:44.814 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f4a24700 1 -- 172.21.15.119:0/301480464 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e800b910 con 0x7f83f006cd70 2022-11-27T17:45:44.815 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.808+0000 7f83f4a24700 1 -- 172.21.15.119:0/301480464 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800bcc0 con 0x7f83f006cd70 2022-11-27T17:45:44.815 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 -- 172.21.15.119:0/301480464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f006cd70 msgr2=0x7f83f006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.815 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/301480464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f006cd70 0x7f83f006d390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83f006b430 tx=0x7f83e800b330 comp rx=0 tx=0).stop 2022-11-27T17:45:44.815 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 -- 172.21.15.119:0/301480464 shutdown_connections 2022-11-27T17:45:44.816 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/301480464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f006cd70 0x7f83f006d390 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.816 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/301480464 >> v2:172.21.15.99:3300/0 conn(0x7f83f015bee0 0x7f83f006c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.816 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 -- 172.21.15.119:0/301480464 >> 172.21.15.119:0/301480464 conn(0x7f83f00b16c0 msgr2=0x7f83f00cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.816 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 -- 172.21.15.119:0/301480464 shutdown_connections 2022-11-27T17:45:44.816 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.809+0000 7f83f7c8a700 1 -- 172.21.15.119:0/301480464 wait complete. 2022-11-27T17:45:44.819 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.810+0000 7f83f7c8a700 1 Processor -- start 2022-11-27T17:45:44.820 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.810+0000 7f83f7c8a700 1 -- start start 2022-11-27T17:45:44.820 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83f006cd70 0x7f83f00c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.820 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83f015bb10 0x7f83f0153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.820 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:44.821 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00c7be0 con 0x7f83f006cd70 2022-11-27T17:45:44.821 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f0155bf0 con 0x7f83f015bee0 2022-11-27T17:45:44.821 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f7c8a700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f0154370 con 0x7f83f015bb10 2022-11-27T17:45:44.821 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.811+0000 7f83f6227700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.821 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55550/0 (socket says 172.21.15.119:55550) 2022-11-27T17:45:44.822 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 -- 172.21.15.119:0/672933177 learned_addr learned my addr 172.21.15.119:0/672933177 (peer_addr_for_me v2:172.21.15.119:0/0) 2022-11-27T17:45:44.822 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f5a26700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83f006cd70 0x7f83f00c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.822 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f5225700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83f015bb10 0x7f83f0153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:44.823 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 -- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83f015bb10 msgr2=0x7f83f0153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.824 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83f015bb10 0x7f83f0153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.825 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 -- 172.21.15.119:0/672933177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83f006cd70 msgr2=0x7f83f00c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.825 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83f006cd70 0x7f83f00c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.825 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 -- 172.21.15.119:0/672933177 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8009b30 con 0x7f83f015bee0 2022-11-27T17:45:44.826 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f6227700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83f006a060 tx=0x7f83e8004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:44.826 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8004880 con 0x7f83f015bee0 2022-11-27T17:45:44.826 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e80092e0 con 0x7f83f015bee0 2022-11-27T17:45:44.826 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f00c7340 con 0x7f83f015bee0 2022-11-27T17:45:44.827 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e801c660 con 0x7f83f015bee0 2022-11-27T17:45:44.827 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f00c7900 con 0x7f83f015bee0 2022-11-27T17:45:44.827 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f83e8022030 con 0x7f83f015bee0 2022-11-27T17:45:44.828 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.812+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f83e801ce00 con 0x7f83f015bee0 2022-11-27T17:45:44.828 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.813+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f0155950 con 0x7f83f015bee0 2022-11-27T17:45:44.828 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.815+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7f83f0155950 con 0x7f83f015bee0 2022-11-27T17:45:44.931 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.929+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "48b889c7-f370-4749-bb57-a26ebf4eec3c", "id": 7} v 0) v1 -- 0x7f83f0065ac0 con 0x7f83f015bee0 2022-11-27T17:45:44.939 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.938+0000 7f83de7fc700 1 -- 172.21.15.119:0/672933177 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48b889c7-f370-4749-bb57-a26ebf4eec3c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f83e8020070 con 0x7f83f015bee0 2022-11-27T17:45:44.940 INFO:teuthology.orchestra.run.smithi119.stdout:7 2022-11-27T17:45:44.941 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.938+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 msgr2=0x7f83f00c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:44.941 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.938+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83f006a060 tx=0x7f83e8004ad0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.942 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 shutdown_connections 2022-11-27T17:45:44.942 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83f006cd70 0x7f83f00c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.943 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f015bee0 0x7f83f00c84d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.943 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 --2- 172.21.15.119:0/672933177 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83f015bb10 0x7f83f0153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:44.943 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 >> 172.21.15.119:0/672933177 conn(0x7f83f00b16c0 msgr2=0x7f83f00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:44.943 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 shutdown_connections 2022-11-27T17:45:44.944 INFO:teuthology.orchestra.run.smithi119.stderr:2022-11-27T17:45:44.939+0000 7f83f7c8a700 1 -- 172.21.15.119:0/672933177 wait complete. 2022-11-27T17:45:44.952 INFO:tasks.ceph.osd.0:Restarting daemon 2022-11-27T17:45:44.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-11-27T17:45:44.955 INFO:tasks.ceph.osd.0:Started 2022-11-27T17:45:44.956 INFO:tasks.ceph.osd.1:Restarting daemon 2022-11-27T17:45:44.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-11-27T17:45:44.958 INFO:tasks.ceph.osd.1:Started 2022-11-27T17:45:44.959 INFO:tasks.ceph.osd.2:Restarting daemon 2022-11-27T17:45:44.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-11-27T17:45:44.963 INFO:tasks.ceph.osd.2:Started 2022-11-27T17:45:44.964 INFO:tasks.ceph.osd.3:Restarting daemon 2022-11-27T17:45:44.964 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-11-27T17:45:44.966 INFO:tasks.ceph.osd.3:Started 2022-11-27T17:45:44.966 INFO:tasks.ceph.osd.4:Restarting daemon 2022-11-27T17:45:44.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-11-27T17:45:44.969 INFO:tasks.ceph.osd.4:Started 2022-11-27T17:45:44.970 INFO:tasks.ceph.osd.5:Restarting daemon 2022-11-27T17:45:44.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-11-27T17:45:44.972 INFO:tasks.ceph.osd.5:Started 2022-11-27T17:45:44.973 INFO:tasks.ceph.osd.6:Restarting daemon 2022-11-27T17:45:44.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-11-27T17:45:44.975 INFO:tasks.ceph.osd.6:Started 2022-11-27T17:45:44.976 INFO:tasks.ceph.osd.7:Restarting daemon 2022-11-27T17:45:44.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-11-27T17:45:44.979 INFO:tasks.ceph.osd.7:Started 2022-11-27T17:45:44.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-27T17:45:45.021 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:45.019+0000 7f504b13c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-11-27T17:45:45.129 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.127+0000 7fc85a22e700 1 Processor -- start 2022-11-27T17:45:45.130 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.127+0000 7fc85a22e700 1 -- start start 2022-11-27T17:45:45.136 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc85a22e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc854158f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:45.137 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc85a22e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc85406f1d0 0x7fc85406f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:45.137 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc85a22e700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc85415bc70 con 0x7fc8540ca470 2022-11-27T17:45:45.137 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc85a22e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8540c9f80 con 0x7fc85406f1d0 2022-11-27T17:45:45.138 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc85a22e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85406fb90 con 0x7fc8541523f0 2022-11-27T17:45:45.138 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc852ffd700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc8540ca470 0x7fc85415c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52636/0 (socket says 172.21.15.99:52636) 2022-11-27T17:45:45.138 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc852ffd700 1 -- 172.21.15.99:0/2381913040 learned_addr learned my addr 172.21.15.99:0/2381913040 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:45:45.139 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc8537fe700 1 --2- 172.21.15.99:0/2381913040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc854158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:45.139 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.128+0000 7fc853fff700 1 --2- 172.21.15.99:0/2381913040 >> v2:172.21.15.119:3300/0 conn(0x7fc85406f1d0 0x7fc85406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:45.139 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8527fc700 1 -- 172.21.15.99:0/2381913040 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162982049 0 0) 0x7fc85415bc70 con 0x7fc8540ca470 2022-11-27T17:45:45.139 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8527fc700 1 -- 172.21.15.99:0/2381913040 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8400035b0 con 0x7fc8540ca470 2022-11-27T17:45:45.140 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8537fe700 1 -- 172.21.15.99:0/2381913040 >> v2:172.21.15.119:3300/0 conn(0x7fc85406f1d0 msgr2=0x7fc85406f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:45.140 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8537fe700 1 --2- 172.21.15.99:0/2381913040 >> v2:172.21.15.119:3300/0 conn(0x7fc85406f1d0 0x7fc85406f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.140 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8537fe700 1 -- 172.21.15.99:0/2381913040 >> v1:172.21.15.99:6789/0 conn(0x7fc8540ca470 legacy=0x7fc85415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:45.140 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8537fe700 1 -- 172.21.15.99:0/2381913040 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540c8800 con 0x7fc8541523f0 2022-11-27T17:45:45.141 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8537fe700 1 --2- 172.21.15.99:0/2381913040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc854158f10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d90 tx=0x7fc84400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c37c532d209d0fe server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:45.141 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8527fc700 1 -- 172.21.15.99:0/2381913040 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84400e070 con 0x7fc8541523f0 2022-11-27T17:45:45.141 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8527fc700 1 -- 172.21.15.99:0/2381913040 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84400bb20 con 0x7fc8541523f0 2022-11-27T17:45:45.141 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.129+0000 7fc8527fc700 1 -- 172.21.15.99:0/2381913040 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844012430 con 0x7fc8541523f0 2022-11-27T17:45:45.142 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 -- 172.21.15.99:0/2381913040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 msgr2=0x7fc854158f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:45.142 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 --2- 172.21.15.99:0/2381913040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc854158f10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc844009d90 tx=0x7fc84400b330 comp rx=0 tx=0).stop 2022-11-27T17:45:45.142 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 -- 172.21.15.99:0/2381913040 shutdown_connections 2022-11-27T17:45:45.143 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 --2- 172.21.15.99:0/2381913040 >> v2:172.21.15.119:3300/0 conn(0x7fc85406f1d0 0x7fc85406f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.143 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 --2- 172.21.15.99:0/2381913040 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc854158f10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.143 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 -- 172.21.15.99:0/2381913040 >> 172.21.15.99:0/2381913040 conn(0x7fc8540ceed0 msgr2=0x7fc8540cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:45.144 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 -- 172.21.15.99:0/2381913040 shutdown_connections 2022-11-27T17:45:45.144 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.130+0000 7fc85a22e700 1 -- 172.21.15.99:0/2381913040 wait complete. 2022-11-27T17:45:45.144 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.131+0000 7fc85a22e700 1 Processor -- start 2022-11-27T17:45:45.145 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.131+0000 7fc85a22e700 1 -- start start 2022-11-27T17:45:45.145 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:45.145 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8540ca470 0x7fc8540c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc8540c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540c5470 con 0x7fc85406f1d0 2022-11-27T17:45:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540b3800 con 0x7fc8540ca470 2022-11-27T17:45:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc85a22e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8540b3ae0 con 0x7fc8541523f0 2022-11-27T17:45:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38388/0 (socket says 172.21.15.99:38388) 2022-11-27T17:45:45.148 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 -- 172.21.15.99:0/3212891386 learned_addr learned my addr 172.21.15.99:0/3212891386 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:45:45.148 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc852ffd700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8540ca470 0x7fc8540c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:45.148 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc853fff700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc8540c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:45.148 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 -- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 msgr2=0x7fc8540c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:45.149 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc8540c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.149 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 -- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8540ca470 msgr2=0x7fc8540c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:45.149 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8540ca470 0x7fc8540c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.149 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 -- 172.21.15.99:0/3212891386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009a70 con 0x7fc85406f1d0 2022-11-27T17:45:45.149 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc8537fe700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc844014040 tx=0x7fc844004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:45.150 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.132+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844012430 con 0x7fc85406f1d0 2022-11-27T17:45:45.150 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc844020070 con 0x7fc85406f1d0 2022-11-27T17:45:45.150 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8540b3dc0 con 0x7fc85406f1d0 2022-11-27T17:45:45.150 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844004280 con 0x7fc85406f1d0 2022-11-27T17:45:45.151 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8540c1470 con 0x7fc85406f1d0 2022-11-27T17:45:45.151 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc8440046d0 con 0x7fc85406f1d0 2022-11-27T17:45:45.151 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fc844022070 con 0x7fc85406f1d0 2022-11-27T17:45:45.151 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.133+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8540b3c20 con 0x7fc85406f1d0 2022-11-27T17:45:45.152 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.136+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7fc8540b3c20 con 0x7fc85406f1d0 2022-11-27T17:45:45.183 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:45.181+0000 7f504b13c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-11-27T17:45:45.207 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.204+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc854073030 con 0x7fc85406f1d0 2022-11-27T17:45:45.207 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.204+0000 7fc83bfff700 1 -- 172.21.15.99:0/3212891386 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7011 (secure 0 0 0) 0x7fc844012e40 con 0x7fc85406f1d0 2022-11-27T17:45:45.208 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:45:45.209 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":10,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:45:44.931976+0000","last_up_change":"0.000000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:45:45.211 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 msgr2=0x7fc8540b31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:45.211 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc844014040 tx=0x7fc844004ad0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.212 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 shutdown_connections 2022-11-27T17:45:45.212 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85406f1d0 0x7fc8540b31f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.213 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8541523f0 0x7fc8540c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.213 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 --2- 172.21.15.99:0/3212891386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8540ca470 0x7fc8540c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:45.213 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.205+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 >> 172.21.15.99:0/3212891386 conn(0x7fc8540ceed0 msgr2=0x7fc854158950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:45.214 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.206+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 shutdown_connections 2022-11-27T17:45:45.214 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:45:45.206+0000 7fc85a22e700 1 -- 172.21.15.99:0/3212891386 wait complete. 2022-11-27T17:45:45.220 INFO:tasks.ceph.ceph_manager.ceph:[] 2022-11-27T17:45:45.221 INFO:tasks.ceph:Waiting for OSDs to come up 2022-11-27T17:45:45.435 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:45.433+0000 7f3491f81200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-11-27T17:45:45.509 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:45.507+0000 7f3491f81200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-11-27T17:45:45.572 INFO:tasks.ceph.osd.4.smithi119.stderr:2022-11-27T17:45:45.570+0000 7f44760c95c0 -1 Falling back to public interface 2022-11-27T17:45:45.573 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:45.570+0000 7fee4b1fa5c0 -1 Falling back to public interface 2022-11-27T17:45:45.573 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:45.570+0000 7fee425b8700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2022-11-27T17:45:45.573 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:45.571+0000 7fee425b8700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6806/0: (98) Address already in use 2022-11-27T17:45:45.589 INFO:tasks.ceph.osd.1.smithi099.stderr:2022-11-27T17:45:45.587+0000 7f9a23d7f5c0 -1 Falling back to public interface 2022-11-27T17:45:45.597 INFO:tasks.ceph.osd.2.smithi099.stderr:2022-11-27T17:45:45.596+0000 7f867df875c0 -1 Falling back to public interface 2022-11-27T17:45:45.600 INFO:tasks.ceph.osd.6.smithi119.stderr:2022-11-27T17:45:45.598+0000 7fb2c34df5c0 -1 Falling back to public interface 2022-11-27T17:45:45.602 INFO:tasks.ceph.osd.0.smithi099.stderr:2022-11-27T17:45:45.600+0000 7f2c2ed275c0 -1 Falling back to public interface 2022-11-27T17:45:45.603 INFO:tasks.ceph.osd.3.smithi099.stderr:2022-11-27T17:45:45.601+0000 7f5a3c1915c0 -1 Falling back to public interface 2022-11-27T17:45:45.609 INFO:tasks.ceph.osd.7.smithi119.stderr:2022-11-27T17:45:45.607+0000 7f4a42a615c0 -1 Falling back to public interface 2022-11-27T17:45:45.649 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:45.647+0000 7f3491f81200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-11-27T17:45:45.803 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:45.801+0000 7f3491f81200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-11-27T17:45:45.857 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:45.855+0000 7f504b13c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-11-27T17:45:45.934 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:45.932+0000 7f504b13c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-11-27T17:45:45.935 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:45.933+0000 7f3491f81200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-11-27T17:45:46.028 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-11-27T17:45:46.069 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.067+0000 7f504b13c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-11-27T17:45:46.142 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.140+0000 7fa192dee700 1 Processor -- start 2022-11-27T17:45:46.143 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.140+0000 7fa192dee700 1 -- start start 2022-11-27T17:45:46.146 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa192dee700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:46.146 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa192dee700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 0x7fa18c0abf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:46.146 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa192dee700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18c0ca1f0 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.146 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa192dee700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c15bd00 con 0x7fa18c15b930 2022-11-27T17:45:46.146 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa192dee700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0caed0 con 0x7fa18c0aba90 2022-11-27T17:45:46.147 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa190b8a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:46.147 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa19138b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 0x7fa18c0abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:46.147 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa190b8a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.99:57158/0 (socket says 172.21.15.99:57158) 2022-11-27T17:45:46.148 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa190b8a700 1 -- 172.21.15.99:0/2543599501 learned_addr learned my addr 172.21.15.99:0/2543599501 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:45:46.148 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa19138b700 1 -- 172.21.15.99:0/2543599501 >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 msgr2=0x7fa18c0693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:45:46.148 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa19138b700 1 --2- 172.21.15.99:0/2543599501 >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.148 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.141+0000 7fa19138b700 1 -- 172.21.15.99:0/2543599501 >> v1:172.21.15.119:6789/0 conn(0x7fa18c0c1bf0 legacy=0x7fa18c069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:46.149 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa19138b700 1 -- 172.21.15.99:0/2543599501 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c0cb1b0 con 0x7fa18c0aba90 2022-11-27T17:45:46.149 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa190b8a700 1 --2- 172.21.15.99:0/2543599501 >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:46.149 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa19138b700 1 --2- 172.21.15.99:0/2543599501 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 0x7fa18c0abf10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa184009d90 tx=0x7fa18400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=227f47be9a81df40 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:46.150 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa18b7fe700 1 -- 172.21.15.99:0/2543599501 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18400e070 con 0x7fa18c0aba90 2022-11-27T17:45:46.150 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa18b7fe700 1 -- 172.21.15.99:0/2543599501 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18400bb20 con 0x7fa18c0aba90 2022-11-27T17:45:46.150 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa18b7fe700 1 -- 172.21.15.99:0/2543599501 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184012430 con 0x7fa18c0aba90 2022-11-27T17:45:46.150 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 -- 172.21.15.99:0/2543599501 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 msgr2=0x7fa18c0abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:46.150 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 --2- 172.21.15.99:0/2543599501 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 0x7fa18c0abf10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa184009d90 tx=0x7fa18400b330 comp rx=0 tx=0).stop 2022-11-27T17:45:46.151 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 -- 172.21.15.99:0/2543599501 shutdown_connections 2022-11-27T17:45:46.151 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 --2- 172.21.15.99:0/2543599501 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0aba90 0x7fa18c0abf10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.151 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 --2- 172.21.15.99:0/2543599501 >> v2:172.21.15.119:3300/0 conn(0x7fa18c15b930 0x7fa18c0693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.152 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.142+0000 7fa192dee700 1 -- 172.21.15.99:0/2543599501 >> 172.21.15.99:0/2543599501 conn(0x7fa18c06b7d0 msgr2=0x7fa18c06bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:46.152 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.143+0000 7fa192dee700 1 -- 172.21.15.99:0/2543599501 shutdown_connections 2022-11-27T17:45:46.152 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.143+0000 7fa192dee700 1 -- 172.21.15.99:0/2543599501 wait complete. 2022-11-27T17:45:46.152 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.144+0000 7fa192dee700 1 Processor -- start 2022-11-27T17:45:46.153 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.144+0000 7fa192dee700 1 -- start start 2022-11-27T17:45:46.156 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:46.156 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 0x7fa18c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:46.156 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c15b930 0x7fa18c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:46.156 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0b4290 con 0x7fa18c0aba90 2022-11-27T17:45:46.157 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0b4060 con 0x7fa18c15b930 2022-11-27T17:45:46.157 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa192dee700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c06f6b0 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.157 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa190b8a700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:46.157 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa190b8a700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38400/0 (socket says 172.21.15.99:38400) 2022-11-27T17:45:46.157 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa190b8a700 1 -- 172.21.15.99:0/354780810 learned_addr learned my addr 172.21.15.99:0/354780810 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:45:46.158 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 0x7fa18c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:46.158 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa19138b700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c15b930 0x7fa18c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:46.158 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 -- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c15b930 msgr2=0x7fa18c0c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:46.158 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c15b930 0x7fa18c0c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.159 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 -- 172.21.15.99:0/354780810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 msgr2=0x7fa18c152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:46.159 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.159 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 -- 172.21.15.99:0/354780810 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009550 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.159 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa190b8a700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:45:46.159 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa18bfff700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 0x7fa18c0b3b20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa178000f60 tx=0x7fa17800c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:46.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1780052d0 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.145+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa178005430 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1780038c0 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa184009a70 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.161 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c0b2560 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.161 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fa178019030 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.161 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fa17800ce30 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.161 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.146+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c06f9e0 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.162 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.149+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71351 (secure 0 0 0) 0x7fa178017070 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.220 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.218+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa18c073030 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.222 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.219+0000 7fa1897fa700 1 -- 172.21.15.99:0/354780810 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7011 (secure 0 0 0) 0x7fa178003a20 con 0x7fa18c0c1bf0 2022-11-27T17:45:46.223 INFO:teuthology.misc.health.smithi099.stdout: 2022-11-27T17:45:46.223 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:45:44.931976+0000","last_up_change":"0.000000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:45:46.226 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 msgr2=0x7fa18c0b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:46.226 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 0x7fa18c0b3b20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa178000f60 tx=0x7fa17800c730 comp rx=0 tx=0).stop 2022-11-27T17:45:46.227 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 shutdown_connections 2022-11-27T17:45:46.227 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa18c0aba90 0x7fa18c152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.228 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c15b930 0x7fa18c0c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.228 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 --2- 172.21.15.99:0/354780810 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa18c0c1bf0 0x7fa18c0b3b20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:46.229 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 >> 172.21.15.99:0/354780810 conn(0x7fa18c06b7d0 msgr2=0x7fa18c159330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:46.229 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 shutdown_connections 2022-11-27T17:45:46.230 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:46.220+0000 7fa192dee700 1 -- 172.21.15.99:0/354780810 wait complete. 2022-11-27T17:45:46.230 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.226+0000 7f504b13c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-11-27T17:45:46.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-11-27T17:45:46.289 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:46.287+0000 7f3491f81200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-11-27T17:45:46.358 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.356+0000 7f504b13c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-11-27T17:45:46.441 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:46.439+0000 7f3491f81200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-11-27T17:45:46.555 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:46.553+0000 7f3491f81200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-11-27T17:45:46.696 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:46.694+0000 7f3491f81200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-11-27T17:45:46.706 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.705+0000 7f504b13c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-11-27T17:45:46.795 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:46.793+0000 7f3491f81200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-11-27T17:45:46.857 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.855+0000 7f504b13c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-11-27T17:45:46.971 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:46.969+0000 7f504b13c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-11-27T17:45:47.117 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:47.115+0000 7f504b13c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-11-27T17:45:47.144 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:47.142+0000 7f3491f81200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-11-27T17:45:47.213 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:47.211+0000 7f504b13c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-11-27T17:45:47.567 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:47.565+0000 7f504b13c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-11-27T17:45:47.719 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:47.717+0000 7f3491f81200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-11-27T17:45:48.150 INFO:tasks.ceph.mgr.x.smithi119.stderr:2022-11-27T17:45:48.148+0000 7f504b13c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-11-27T17:45:48.742 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:48.740+0000 7f344a34b700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-11-27T17:45:48.742 INFO:tasks.ceph.mgr.y.smithi099.stderr:2022-11-27T17:45:48.740+0000 7f344a34b700 -1 mgr.server handle_report got status from non-daemon mon.c 2022-11-27T17:45:49.123 INFO:tasks.ceph.osd.1.smithi099.stderr:2022-11-27T17:45:49.121+0000 7f9a23d7f5c0 -1 osd.1 0 log_to_monitors true 2022-11-27T17:45:49.150 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:49.149+0000 7fee4b1fa5c0 -1 osd.5 0 log_to_monitors true 2022-11-27T17:45:49.158 INFO:tasks.ceph.osd.4.smithi119.stderr:2022-11-27T17:45:49.157+0000 7f44760c95c0 -1 osd.4 0 log_to_monitors true 2022-11-27T17:45:49.167 INFO:tasks.ceph.osd.3.smithi099.stderr:2022-11-27T17:45:49.165+0000 7f5a3c1915c0 -1 osd.3 0 log_to_monitors true 2022-11-27T17:45:49.205 INFO:tasks.ceph.osd.2.smithi099.stderr:2022-11-27T17:45:49.203+0000 7f867df875c0 -1 osd.2 0 log_to_monitors true 2022-11-27T17:45:49.225 INFO:tasks.ceph.osd.0.smithi099.stderr:2022-11-27T17:45:49.223+0000 7f2c2ed275c0 -1 osd.0 0 log_to_monitors true 2022-11-27T17:45:49.226 INFO:tasks.ceph.osd.6.smithi119.stderr:2022-11-27T17:45:49.224+0000 7fb2c34df5c0 -1 osd.6 0 log_to_monitors true 2022-11-27T17:45:49.232 INFO:tasks.ceph.osd.7.smithi119.stderr:2022-11-27T17:45:49.230+0000 7f4a42a615c0 -1 osd.7 0 log_to_monitors true 2022-11-27T17:45:52.750 INFO:tasks.ceph.osd.3.smithi099.stderr:2022-11-27T17:45:52.748+0000 7f5a2fd48700 -1 osd.3 0 waiting for initial osdmap 2022-11-27T17:45:52.755 INFO:tasks.ceph.osd.0.smithi099.stderr:2022-11-27T17:45:52.753+0000 7f2c228de700 -1 osd.0 0 waiting for initial osdmap 2022-11-27T17:45:52.767 INFO:tasks.ceph.osd.1.smithi099.stderr:2022-11-27T17:45:52.764+0000 7f9a17936700 -1 osd.1 0 waiting for initial osdmap 2022-11-27T17:45:52.768 INFO:tasks.ceph.osd.3.smithi099.stderr:2022-11-27T17:45:52.767+0000 7f5a28123700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.771 INFO:tasks.ceph.osd.0.smithi099.stderr:2022-11-27T17:45:52.770+0000 7f2c1acb9700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.779 INFO:tasks.ceph.osd.1.smithi099.stderr:2022-11-27T17:45:52.778+0000 7f9a0fd11700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.805 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:52.804+0000 7fee3edb1700 -1 osd.5 0 waiting for initial osdmap 2022-11-27T17:45:52.807 INFO:tasks.ceph.osd.2.smithi099.stderr:2022-11-27T17:45:52.805+0000 7f8671b3e700 -1 osd.2 0 waiting for initial osdmap 2022-11-27T17:45:52.826 INFO:tasks.ceph.osd.7.smithi119.stderr:2022-11-27T17:45:52.824+0000 7f4a36618700 -1 osd.7 0 waiting for initial osdmap 2022-11-27T17:45:52.827 INFO:tasks.ceph.osd.2.smithi099.stderr:2022-11-27T17:45:52.826+0000 7f8669f19700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.828 INFO:tasks.ceph.osd.5.smithi119.stderr:2022-11-27T17:45:52.827+0000 7fee3718c700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.832 INFO:tasks.ceph.osd.6.smithi119.stderr:2022-11-27T17:45:52.831+0000 7fb2b7096700 -1 osd.6 0 waiting for initial osdmap 2022-11-27T17:45:52.838 INFO:tasks.ceph.osd.7.smithi119.stderr:2022-11-27T17:45:52.837+0000 7f4a2e9f3700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.842 INFO:tasks.ceph.osd.4.smithi119.stderr:2022-11-27T17:45:52.841+0000 7f4469c80700 -1 osd.4 0 waiting for initial osdmap 2022-11-27T17:45:52.846 INFO:tasks.ceph.osd.6.smithi119.stderr:2022-11-27T17:45:52.845+0000 7fb2af471700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:52.855 INFO:tasks.ceph.osd.4.smithi119.stderr:2022-11-27T17:45:52.854+0000 7f446205b700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-11-27T17:45:53.039 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-11-27T17:45:53.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.158+0000 7f3574bee700 1 Processor -- start 2022-11-27T17:45:53.160 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.158+0000 7f3574bee700 1 -- start start 2022-11-27T17:45:53.163 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f3574bee700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 0x7f357006a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:53.163 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f3574bee700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357006ae90 con 0x7f357006aa40 2022-11-27T17:45:53.163 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f3574bee700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700cdb60 con 0x7f35700cbbd0 2022-11-27T17:45:53.163 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f3574bee700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700cc050 con 0x7f35700b21c0 2022-11-27T17:45:53.164 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f356dd9b700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f357006aa40 0x7f35700cb4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52758/0 (socket says 172.21.15.99:52758) 2022-11-27T17:45:53.164 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f356dd9b700 1 -- 172.21.15.99:0/964947474 learned_addr learned my addr 172.21.15.99:0/964947474 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:45:53.164 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.159+0000 7f356e59c700 1 --2- 172.21.15.99:0/964947474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 0x7f357006a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:53.164 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356e59c700 1 -- 172.21.15.99:0/964947474 >> v1:172.21.15.119:6789/0 conn(0x7f35700cbbd0 legacy=0x7f35700ce890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.164 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356e59c700 1 -- 172.21.15.99:0/964947474 >> v1:172.21.15.99:6789/0 conn(0x7f357006aa40 legacy=0x7f35700cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.165 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356e59c700 1 -- 172.21.15.99:0/964947474 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357015c030 con 0x7f35700b21c0 2022-11-27T17:45:53.165 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356e59c700 1 --2- 172.21.15.99:0/964947474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 0x7f357006a500 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3560009870 tx=0x7f3560009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef49de22f809e6b4 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:53.165 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356d59a700 1 -- 172.21.15.99:0/964947474 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560011830 con 0x7f35700b21c0 2022-11-27T17:45:53.165 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356d59a700 1 -- 172.21.15.99:0/964947474 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560011990 con 0x7f35700b21c0 2022-11-27T17:45:53.165 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.160+0000 7f356d59a700 1 -- 172.21.15.99:0/964947474 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560011d20 con 0x7f35700b21c0 2022-11-27T17:45:53.166 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 -- 172.21.15.99:0/964947474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 msgr2=0x7f357006a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.166 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 --2- 172.21.15.99:0/964947474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 0x7f357006a500 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3560009870 tx=0x7f3560009b80 comp rx=0 tx=0).stop 2022-11-27T17:45:53.166 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 -- 172.21.15.99:0/964947474 shutdown_connections 2022-11-27T17:45:53.166 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 --2- 172.21.15.99:0/964947474 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700b21c0 0x7f357006a500 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.166 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 -- 172.21.15.99:0/964947474 >> 172.21.15.99:0/964947474 conn(0x7f3570069d10 msgr2=0x7f3570099270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:53.167 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 -- 172.21.15.99:0/964947474 shutdown_connections 2022-11-27T17:45:53.167 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.161+0000 7f3574bee700 1 -- 172.21.15.99:0/964947474 wait complete. 2022-11-27T17:45:53.170 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.162+0000 7f3574bee700 1 Processor -- start 2022-11-27T17:45:53.170 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.162+0000 7f3574bee700 1 -- start start 2022-11-27T17:45:53.171 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 0x7f3570154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:53.171 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:53.171 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700cbbd0 0x7f35701537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:53.172 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b3de0 con 0x7f35700b21c0 2022-11-27T17:45:53.172 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700b3410 con 0x7f357006aa40 2022-11-27T17:45:53.172 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f3574bee700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35700c5790 con 0x7f35700cbbd0 2022-11-27T17:45:53.172 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:53.173 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38510/0 (socket says 172.21.15.99:38510) 2022-11-27T17:45:53.173 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 -- 172.21.15.99:0/986600193 learned_addr learned my addr 172.21.15.99:0/986600193 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:45:53.173 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356e59c700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 0x7f3570154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:53.173 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356ed9d700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700cbbd0 0x7f35701537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:53.173 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 -- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700cbbd0 msgr2=0x7f35701537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.174 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700cbbd0 0x7f35701537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.174 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 -- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 msgr2=0x7f3570154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.174 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 0x7f3570154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.174 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 -- 172.21.15.99:0/986600193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35600094a0 con 0x7f35700b21c0 2022-11-27T17:45:53.174 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356e59c700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 0x7f3570154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:45:53.175 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f356dd9b700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f356400c520 tx=0x7f356400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:53.175 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564005d70 con 0x7f35700b21c0 2022-11-27T17:45:53.175 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.163+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35640070d0 con 0x7f35700b21c0 2022-11-27T17:45:53.175 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.164+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35700c5a70 con 0x7f35700b21c0 2022-11-27T17:45:53.176 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.164+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564007440 con 0x7f35700b21c0 2022-11-27T17:45:53.176 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.164+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35700b30a0 con 0x7f35700b21c0 2022-11-27T17:45:53.176 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.165+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570073030 con 0x7f35700b21c0 2022-11-27T17:45:53.176 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.165+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 4) v1 ==== 84457+0+0 (secure 0 0 0) 0x7f3564019030 con 0x7f35700b21c0 2022-11-27T17:45:53.176 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.165+0000 7f355effd700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 0x7f354c068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:45:53.177 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.165+0000 7f356e59c700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 0x7f354c068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:45:53.177 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.165+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f35640819c0 con 0x7f35700b21c0 2022-11-27T17:45:53.177 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.166+0000 7f356e59c700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 0x7f354c068a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3560000c00 tx=0x7f3560008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:45:53.178 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.170+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f356404abd0 con 0x7f35700b21c0 2022-11-27T17:45:53.349 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.347+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f357006ae90 con 0x7f35700b21c0 2022-11-27T17:45:53.350 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.347+0000 7f355effd700 1 -- 172.21.15.99:0/986600193 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7011 (secure 0 0 0) 0x7f357006ae90 con 0x7f35700b21c0 2022-11-27T17:45:53.350 INFO:teuthology.misc.health.smithi099.stdout: 2022-11-27T17:45:53.350 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":12,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:45:50.738227+0000","last_up_change":"0.000000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:45:53.352 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 msgr2=0x7f354c068a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.352 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 0x7f354c068a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3560000c00 tx=0x7f3560008030 comp rx=0 tx=0).stop 2022-11-27T17:45:53.353 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 msgr2=0x7f35700b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:45:53.353 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f356400c520 tx=0x7f356400c8e0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.353 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 shutdown_connections 2022-11-27T17:45:53.353 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f35700cbbd0 0x7f35701537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.354 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f357006aa40 0x7f3570154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.354 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f354c066570 0x7f354c068a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.354 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 --2- 172.21.15.99:0/986600193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35700b21c0 0x7f35700b38a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:45:53.354 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 >> 172.21.15.99:0/986600193 conn(0x7f3570069d10 msgr2=0x7f357006d200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:45:53.355 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 shutdown_connections 2022-11-27T17:45:53.355 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:45:53.350+0000 7f3574bee700 1 -- 172.21.15.99:0/986600193 wait complete. 2022-11-27T17:45:53.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-11-27T17:46:00.171 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-11-27T17:46:00.294 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.293+0000 7f20c8ab2700 1 Processor -- start 2022-11-27T17:46:00.295 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.293+0000 7f20c8ab2700 1 -- start start 2022-11-27T17:46:00.306 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c8ab2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c41598e0 0x7f20c40af830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.307 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c8ab2700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40b2de0 con 0x7f20c40adc00 2022-11-27T17:46:00.307 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c8ab2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40b0180 con 0x7f20c40ce270 2022-11-27T17:46:00.307 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c8ab2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b0460 con 0x7f20c41598e0 2022-11-27T17:46:00.308 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c1d9b700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f20c40adc00 0x7f20c40cdb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52992/0 (socket says 172.21.15.99:52992) 2022-11-27T17:46:00.308 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c1d9b700 1 -- 172.21.15.99:0/1182650073 learned_addr learned my addr 172.21.15.99:0/1182650073 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:00.308 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c259c700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c41598e0 0x7f20c40af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.308 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040486940 0 0) 0x7f20c40b2de0 con 0x7f20c40adc00 2022-11-27T17:46:00.309 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a80035b0 con 0x7f20c40adc00 2022-11-27T17:46:00.309 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957322328 0 0) 0x7f20c40b0180 con 0x7f20c40ce270 2022-11-27T17:46:00.309 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.294+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c40b2de0 con 0x7f20c40ce270 2022-11-27T17:46:00.309 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510591368 0 0) 0x7f20a80035b0 con 0x7f20c40adc00 2022-11-27T17:46:00.309 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c41598e0 msgr2=0x7f20c40af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.310 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c41598e0 0x7f20c40af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.310 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 >> v1:172.21.15.119:6789/0 conn(0x7f20c40ce270 legacy=0x7f20c40b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.310 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40b0740 con 0x7f20c40adc00 2022-11-27T17:46:00.311 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2447476370 0 0) 0x7f20b4001670 con 0x7f20c40adc00 2022-11-27T17:46:00.311 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 >> v1:172.21.15.99:6789/0 conn(0x7f20c40adc00 legacy=0x7f20c40cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.312 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8003bc0 0x7f20a8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.312 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20a8004850 0x7f20a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.313 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20a8005200 0x7f20a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.313 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b0460 con 0x7f20a8005200 2022-11-27T17:46:00.313 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a800bfd0 con 0x7f20a8004850 2022-11-27T17:46:00.314 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a800c190 con 0x7f20a8003bc0 2022-11-27T17:46:00.314 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c259c700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20c41598e0 0x7f20c40af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:46:00.315 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20a8005200 0x7f20a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.315 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c1d9b700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20a8004850 0x7f20a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.316 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c259c700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8003bc0 0x7f20a8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.316 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 -- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8003bc0 msgr2=0x7f20a8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.317 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8003bc0 0x7f20a8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.317 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 -- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20a8004850 msgr2=0x7f20a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.318 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20a8004850 0x7f20a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.318 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 -- 172.21.15.99:0/1182650073 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80035b0 con 0x7f20a8005200 2022-11-27T17:46:00.319 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20a8005200 0x7f20a800ba10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cff60 tx=0x7f20a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07667403912ee68 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.319 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc001f20 con 0x7f20a8005200 2022-11-27T17:46:00.320 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc00ce40 con 0x7f20a8005200 2022-11-27T17:46:00.320 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.295+0000 7f20c159a700 1 -- 172.21.15.99:0/1182650073 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc003c40 con 0x7f20a8005200 2022-11-27T17:46:00.321 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 -- 172.21.15.99:0/1182650073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20a8005200 msgr2=0x7f20a800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.321 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/1182650073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20a8005200 0x7f20a800ba10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cff60 tx=0x7f20a8005880 comp rx=0 tx=0).stop 2022-11-27T17:46:00.322 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c2d9d700 1 -- 172.21.15.99:0/1182650073 reap_dead start 2022-11-27T17:46:00.322 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 -- 172.21.15.99:0/1182650073 shutdown_connections 2022-11-27T17:46:00.322 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 -- 172.21.15.99:0/1182650073 >> 172.21.15.99:0/1182650073 conn(0x7f20c4069d10 msgr2=0x7f20c4099270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:00.322 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 -- 172.21.15.99:0/1182650073 shutdown_connections 2022-11-27T17:46:00.323 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.296+0000 7f20c8ab2700 1 -- 172.21.15.99:0/1182650073 wait complete. 2022-11-27T17:46:00.323 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.297+0000 7f20c8ab2700 1 Processor -- start 2022-11-27T17:46:00.323 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.297+0000 7f20c8ab2700 1 -- start start 2022-11-27T17:46:00.324 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8005200 0x7f20c40cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.324 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.324 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 0x7f20c40b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.324 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c414ca40 con 0x7f20c415c3e0 2022-11-27T17:46:00.324 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40cc3d0 con 0x7f20c406aa40 2022-11-27T17:46:00.325 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c8ab2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40ccda0 con 0x7f20a8005200 2022-11-27T17:46:00.325 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c1d9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.325 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c1d9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38750/0 (socket says 172.21.15.99:38750) 2022-11-27T17:46:00.325 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c1d9b700 1 -- 172.21.15.99:0/766803379 learned_addr learned my addr 172.21.15.99:0/766803379 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:00.326 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c259c700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8005200 0x7f20c40cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.326 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.298+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 0x7f20c40b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.326 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 -- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8005200 msgr2=0x7f20c40cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.326 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8005200 0x7f20c40cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.327 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 -- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 msgr2=0x7f20c40b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.327 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 0x7f20c40b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.327 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 -- 172.21.15.99:0/766803379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40cb7c0 con 0x7f20c415c3e0 2022-11-27T17:46:00.327 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c2d9d700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 0x7f20c40b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-11-27T17:46:00.328 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c1d9b700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20b4007340 tx=0x7f20b4007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.328 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4007820 con 0x7f20c415c3e0 2022-11-27T17:46:00.328 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b400f870 con 0x7f20c415c3e0 2022-11-27T17:46:00.328 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b400fc20 con 0x7f20c415c3e0 2022-11-27T17:46:00.329 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c40cbaa0 con 0x7f20c415c3e0 2022-11-27T17:46:00.329 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.299+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c40cc060 con 0x7f20c415c3e0 2022-11-27T17:46:00.329 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.300+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c40cc220 con 0x7f20c415c3e0 2022-11-27T17:46:00.329 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.300+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f20b400fd80 con 0x7f20c415c3e0 2022-11-27T17:46:00.330 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.300+0000 7f20b2ffd700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 0x7f20a0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.330 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.301+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f20b4083440 con 0x7f20c415c3e0 2022-11-27T17:46:00.330 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.301+0000 7f20c259c700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 0x7f20a0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.330 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.301+0000 7f20c259c700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 0x7f20a0068a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20c4070ec0 tx=0x7f20b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.330 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.305+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f20b4051000 con 0x7f20c415c3e0 2022-11-27T17:46:00.483 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.482+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20c4073030 con 0x7f20c415c3e0 2022-11-27T17:46:00.484 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.483+0000 7f20b2ffd700 1 -- 172.21.15.99:0/766803379 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12221 (secure 0 0 0) 0x7f20b4054620 con 0x7f20c415c3e0 2022-11-27T17:46:00.485 INFO:teuthology.misc.health.smithi099.stdout: 2022-11-27T17:46:00.485 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":17,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:45:58.223157+0000","last_up_change":"2022-11-27T17:45:53.769684+0000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-27T17:45:55.780809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":101360},{"type":"v1","addr":"172.21.15.99:6817","nonce":101360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":101360},{"type":"v1","addr":"172.21.15.99:6820","nonce":101360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":101360},{"type":"v1","addr":"172.21.15.99:6828","nonce":101360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":101360},{"type":"v1","addr":"172.21.15.99:6824","nonce":101360}]},"public_addr":"172.21.15.99:6817/101360","cluster_addr":"172.21.15.99:6820/101360","heartbeat_back_addr":"172.21.15.99:6828/101360","heartbeat_front_addr":"172.21.15.99:6824/101360","state":["exists","up"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":101359},{"type":"v1","addr":"172.21.15.99:6801","nonce":101359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":101359},{"type":"v1","addr":"172.21.15.99:6803","nonce":101359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":101359},{"type":"v1","addr":"172.21.15.99:6807","nonce":101359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":101359},{"type":"v1","addr":"172.21.15.99:6805","nonce":101359}]},"public_addr":"172.21.15.99:6801/101359","cluster_addr":"172.21.15.99:6803/101359","heartbeat_back_addr":"172.21.15.99:6807/101359","heartbeat_front_addr":"172.21.15.99:6805/101359","state":["exists","up"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":101358},{"type":"v1","addr":"172.21.15.99:6809","nonce":101358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":101358},{"type":"v1","addr":"172.21.15.99:6811","nonce":101358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":101358},{"type":"v1","addr":"172.21.15.99:6815","nonce":101358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":101358},{"type":"v1","addr":"172.21.15.99:6813","nonce":101358}]},"public_addr":"172.21.15.99:6809/101358","cluster_addr":"172.21.15.99:6811/101358","heartbeat_back_addr":"172.21.15.99:6815/101358","heartbeat_front_addr":"172.21.15.99:6813/101358","state":["exists","up"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":101361},{"type":"v1","addr":"172.21.15.99:6821","nonce":101361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6823","nonce":101361},{"type":"v1","addr":"172.21.15.99:6825","nonce":101361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":101361},{"type":"v1","addr":"172.21.15.99:6831","nonce":101361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6827","nonce":101361},{"type":"v1","addr":"172.21.15.99:6829","nonce":101361}]},"public_addr":"172.21.15.99:6821/101361","cluster_addr":"172.21.15.99:6825/101361","heartbeat_back_addr":"172.21.15.99:6831/101361","heartbeat_front_addr":"172.21.15.99:6829/101361","state":["exists","up"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":101614},{"type":"v1","addr":"172.21.15.119:6801","nonce":101614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":101614},{"type":"v1","addr":"172.21.15.119:6805","nonce":101614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":101614},{"type":"v1","addr":"172.21.15.119:6812","nonce":101614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":101614},{"type":"v1","addr":"172.21.15.119:6808","nonce":101614}]},"public_addr":"172.21.15.119:6801/101614","cluster_addr":"172.21.15.119:6805/101614","heartbeat_back_addr":"172.21.15.119:6812/101614","heartbeat_front_addr":"172.21.15.119:6808/101614","state":["exists","up"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":101615},{"type":"v1","addr":"172.21.15.119:6804","nonce":101615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":101615},{"type":"v1","addr":"172.21.15.119:6809","nonce":101615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":101615},{"type":"v1","addr":"172.21.15.119:6815","nonce":101615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":101615},{"type":"v1","addr":"172.21.15.119:6813","nonce":101615}]},"public_addr":"172.21.15.119:6804/101615","cluster_addr":"172.21.15.119:6809/101615","heartbeat_back_addr":"172.21.15.119:6815/101615","heartbeat_front_addr":"172.21.15.119:6813/101615","state":["exists","up"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":101644},{"type":"v1","addr":"172.21.15.119:6817","nonce":101644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":101644},{"type":"v1","addr":"172.21.15.119:6819","nonce":101644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":101644},{"type":"v1","addr":"172.21.15.119:6823","nonce":101644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":101644},{"type":"v1","addr":"172.21.15.119:6821","nonce":101644}]},"public_addr":"172.21.15.119:6817/101644","cluster_addr":"172.21.15.119:6819/101644","heartbeat_back_addr":"172.21.15.119:6823/101644","heartbeat_front_addr":"172.21.15.119:6821/101644","state":["exists","up"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":101645},{"type":"v1","addr":"172.21.15.119:6825","nonce":101645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":101645},{"type":"v1","addr":"172.21.15.119:6827","nonce":101645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":101645},{"type":"v1","addr":"172.21.15.119:6831","nonce":101645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":101645},{"type":"v1","addr":"172.21.15.119:6829","nonce":101645}]},"public_addr":"172.21.15.119:6825/101645","cluster_addr":"172.21.15.119:6827/101645","heartbeat_back_addr":"172.21.15.119:6831/101645","heartbeat_front_addr":"172.21.15.119:6829/101645","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.255830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.201490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.145828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.190338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.159710+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.218916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.273223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:46:00.487 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 msgr2=0x7f20a0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.489 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 0x7f20a0068a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f20c4070ec0 tx=0x7f20b8008040 comp rx=0 tx=0).stop 2022-11-27T17:46:00.489 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 msgr2=0x7f20c406a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.489 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20b4007340 tx=0x7f20b4007700 comp rx=0 tx=0).stop 2022-11-27T17:46:00.489 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 shutdown_connections 2022-11-27T17:46:00.490 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20c406aa40 0x7f20c40b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.490 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f20a8005200 0x7f20c40cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.490 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f20a0066590 0x7f20a0068a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.490 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 --2- 172.21.15.99:0/766803379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20c415c3e0 0x7f20c406a500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.491 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 >> 172.21.15.99:0/766803379 conn(0x7f20c4069d10 msgr2=0x7f20c40c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:00.491 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 shutdown_connections 2022-11-27T17:46:00.491 INFO:teuthology.misc.health.smithi099.stderr:2022-11-27T17:46:00.485+0000 7f20c8ab2700 1 -- 172.21.15.99:0/766803379 wait complete. 2022-11-27T17:46:00.498 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-11-27T17:46:00.499 INFO:tasks.ceph:Creating RBD pool 2022-11-27T17:46:00.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd pool create rbd 8 2022-11-27T17:46:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.617+0000 7fb6a7d35700 1 Processor -- start 2022-11-27T17:46:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 -- start start 2022-11-27T17:46:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb6a006ff80 0x7fb6a0070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb6a00ce510 0x7fb6a006c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a0070890 con 0x7fb6a00c4620 2022-11-27T17:46:00.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a00c4b00 con 0x7fb6a00ce510 2022-11-27T17:46:00.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.618+0000 7fb6a7d35700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a00cda30 con 0x7fb6a006ff80 2022-11-27T17:46:00.627 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb6a00ce510 0x7fb6a006c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb6a00ce510 0x7fb6a006c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38760/0 (socket says 172.21.15.99:38760) 2022-11-27T17:46:00.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 -- 172.21.15.99:0/2255009735 learned_addr learned my addr 172.21.15.99:0/2255009735 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:00.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a5ad1700 1 --2- 172.21.15.99:0/2255009735 >> v2:172.21.15.119:3300/0 conn(0x7fb6a006ff80 0x7fb6a0070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.628 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 -- 172.21.15.99:0/2255009735 >> v2:172.21.15.119:3300/0 conn(0x7fb6a006ff80 msgr2=0x7fb6a0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 --2- 172.21.15.99:0/2255009735 >> v2:172.21.15.119:3300/0 conn(0x7fb6a006ff80 0x7fb6a0070350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 -- 172.21.15.99:0/2255009735 >> v1:172.21.15.119:6789/0 conn(0x7fb6a00c4620 legacy=0x7fb6a00c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 -- 172.21.15.99:0/2255009735 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00cdd10 con 0x7fb6a00ce510 2022-11-27T17:46:00.629 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a62d2700 1 --2- 172.21.15.99:0/2255009735 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a00ce510 0x7fb6a006c050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00cb870 tx=0x7fb69800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e2c286931a793c server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a4acf700 1 -- 172.21.15.99:0/2255009735 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb698010070 con 0x7fb6a00ce510 2022-11-27T17:46:00.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a4acf700 1 -- 172.21.15.99:0/2255009735 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69800dd50 con 0x7fb6a00ce510 2022-11-27T17:46:00.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.619+0000 7fb6a4acf700 1 -- 172.21.15.99:0/2255009735 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69800b310 con 0x7fb6a00ce510 2022-11-27T17:46:00.630 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 -- 172.21.15.99:0/2255009735 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a00ce510 msgr2=0x7fb6a006c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/2255009735 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a00ce510 0x7fb6a006c050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00cb870 tx=0x7fb69800d430 comp rx=0 tx=0).stop 2022-11-27T17:46:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 -- 172.21.15.99:0/2255009735 shutdown_connections 2022-11-27T17:46:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/2255009735 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a00ce510 0x7fb6a006c050 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/2255009735 >> v2:172.21.15.119:3300/0 conn(0x7fb6a006ff80 0x7fb6a0070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 -- 172.21.15.99:0/2255009735 >> 172.21.15.99:0/2255009735 conn(0x7fb6a00b16c0 msgr2=0x7fb6a00cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:00.632 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 -- 172.21.15.99:0/2255009735 shutdown_connections 2022-11-27T17:46:00.632 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.620+0000 7fb6a7d35700 1 -- 172.21.15.99:0/2255009735 wait complete. 2022-11-27T17:46:00.632 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.621+0000 7fb6a7d35700 1 Processor -- start 2022-11-27T17:46:00.632 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.621+0000 7fb6a7d35700 1 -- start start 2022-11-27T17:46:00.633 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.633 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6a00c4620 0x7fb6a0150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.633 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6a00ce510 0x7fb6a0155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.633 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a01596b0 con 0x7fb6a006ff80 2022-11-27T17:46:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0158e10 con 0x7fb6a00c4620 2022-11-27T17:46:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a01590f0 con 0x7fb6a00ce510 2022-11-27T17:46:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38762/0 (socket says 172.21.15.99:38762) 2022-11-27T17:46:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 -- 172.21.15.99:0/3842142430 learned_addr learned my addr 172.21.15.99:0/3842142430 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:00.635 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 -- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6a00ce510 msgr2=0x7fb6a0155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.635 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a62d2700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6a00ce510 0x7fb6a0155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.636 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6a00ce510 0x7fb6a0155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.636 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 -- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6a00c4620 msgr2=0x7fb6a0150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.636 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6a00c4620 0x7fb6a0150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.637 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 -- 172.21.15.99:0/3842142430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69800bd40 con 0x7fb6a006ff80 2022-11-27T17:46:00.637 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a5ad1700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c7f0 tx=0x7fb69000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690011800 con 0x7fb6a006ff80 2022-11-27T17:46:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb690011960 con 0x7fb6a006ff80 2022-11-27T17:46:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a01593d0 con 0x7fb6a006ff80 2022-11-27T17:46:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690011cf0 con 0x7fb6a006ff80 2022-11-27T17:46:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.622+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a0151010 con 0x7fb6a006ff80 2022-11-27T17:46:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.623+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a015bd60 con 0x7fb6a006ff80 2022-11-27T17:46:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.624+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fb690021070 con 0x7fb6a006ff80 2022-11-27T17:46:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.624+0000 7fb68e7fc700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 0x7fb680068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.624+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fb6900830e0 con 0x7fb6a006ff80 2022-11-27T17:46:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.624+0000 7fb6a52d0700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 0x7fb680068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.624+0000 7fb6a52d0700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 0x7fb680068a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00ca4f0 tx=0x7fb69c007a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.628+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fb6a015bd60 con 0x7fb6a006ff80 2022-11-27T17:46:00.807 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.806+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fb6a015bd60 con 0x7fb6a006ff80 2022-11-27T17:46:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.852+0000 7fb68e7fc700 1 -- 172.21.15.99:0/3842142430 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7fb6a015bd60 con 0x7fb6a006ff80 2022-11-27T17:46:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 msgr2=0x7fb680068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 0x7fb680068a40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00ca4f0 tx=0x7fb69c007a20 comp rx=0 tx=0).stop 2022-11-27T17:46:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 msgr2=0x7fb6a0151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb69000c7f0 tx=0x7fb69000cbb0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 shutdown_connections 2022-11-27T17:46:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb680066590 0x7fb680068a40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb6a006ff80 0x7fb6a0151280 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb6a00ce510 0x7fb6a0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 --2- 172.21.15.99:0/3842142430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6a00c4620 0x7fb6a0150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 >> 172.21.15.99:0/3842142430 conn(0x7fb6a00b16c0 msgr2=0x7fb6a00afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 shutdown_connections 2022-11-27T17:46:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.854+0000 7fb6a7d35700 1 -- 172.21.15.99:0/3842142430 wait complete. 2022-11-27T17:46:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'rbd' created 2022-11-27T17:46:00.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-11-27T17:46:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 Processor -- start 2022-11-27T17:46:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 -- start start 2022-11-27T17:46:00.994 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840b1ca0 con 0x7fdf840b2dc0 2022-11-27T17:46:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840d0cb0 con 0x7fdf841598c0 2022-11-27T17:46:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.987+0000 7fdf8acb4700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840cc830 con 0x7fdf8406d830 2022-11-27T17:46:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf83fff700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdf840b2dc0 0x7fdf8406d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53026/0 (socket says 172.21.15.99:53026) 2022-11-27T17:46:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf83fff700 1 -- 172.21.15.99:0/4246904537 learned_addr learned my addr 172.21.15.99:0/4246904537 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf89251700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314733090 0 0) 0x7fdf840b1ca0 con 0x7fdf840b2dc0 2022-11-27T17:46:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf680035b0 con 0x7fdf840b2dc0 2022-11-27T17:46:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45738070 0 0) 0x7fdf840d0cb0 con 0x7fdf841598c0 2022-11-27T17:46:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf840b1ca0 con 0x7fdf841598c0 2022-11-27T17:46:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163115589 0 0) 0x7fdf680035b0 con 0x7fdf840b2dc0 2022-11-27T17:46:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 msgr2=0x7fdf840b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf840b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 >> v1:172.21.15.119:6789/0 conn(0x7fdf841598c0 legacy=0x7fdf840d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840cc270 con 0x7fdf840b2dc0 2022-11-27T17:46:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2447476370 0 0) 0x7fdf70001670 con 0x7fdf840b2dc0 2022-11-27T17:46:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 >> v1:172.21.15.99:6789/0 conn(0x7fdf840b2dc0 legacy=0x7fdf8406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68003bc0 0x7fdf68003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf68004850 0x7fdf68004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840cc830 con 0x7fdf68004850 2022-11-27T17:46:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6800bac0 con 0x7fdf68003bc0 2022-11-27T17:46:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6800bc80 con 0x7fdf8406d830 2022-11-27T17:46:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf83fff700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf68004850 0x7fdf68004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf89251700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.988+0000 7fdf88a50700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68003bc0 0x7fdf68003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 -- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 msgr2=0x7fdf68005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf8406d830 0x7fdf68005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 -- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68003bc0 msgr2=0x7fdf68003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68003bc0 0x7fdf68003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 -- 172.21.15.99:0/4246904537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680035b0 con 0x7fdf68004850 2022-11-27T17:46:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf88a50700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68003bc0 0x7fdf68003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf83fff700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf68004850 0x7fdf68004cc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c210 tx=0x7fdf68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2142a6bfb898d745 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000cdb0 con 0x7fdf68004850 2022-11-27T17:46:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70001af0 con 0x7fdf68004850 2022-11-27T17:46:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf837fe700 1 -- 172.21.15.99:0/4246904537 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70005ba0 con 0x7fdf68004850 2022-11-27T17:46:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 -- 172.21.15.99:0/4246904537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf68004850 msgr2=0x7fdf68004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/4246904537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf68004850 0x7fdf68004cc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c210 tx=0x7fdf68005880 comp rx=0 tx=0).stop 2022-11-27T17:46:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf89251700 1 -- 172.21.15.99:0/4246904537 reap_dead start 2022-11-27T17:46:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 -- 172.21.15.99:0/4246904537 shutdown_connections 2022-11-27T17:46:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 -- 172.21.15.99:0/4246904537 >> 172.21.15.99:0/4246904537 conn(0x7fdf84071060 msgr2=0x7fdf84071430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 -- 172.21.15.99:0/4246904537 shutdown_connections 2022-11-27T17:46:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.989+0000 7fdf8acb4700 1 -- 172.21.15.99:0/4246904537 wait complete. 2022-11-27T17:46:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 Processor -- start 2022-11-27T17:46:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 -- start start 2022-11-27T17:46:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 0x7fdf840c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840a7d00 0x7fdf84148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841474d0 con 0x7fdf840a7890 2022-11-27T17:46:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84147740 con 0x7fdf68004850 2022-11-27T17:46:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.991+0000 7fdf8acb4700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84147a20 con 0x7fdf840a7d00 2022-11-27T17:46:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf88a50700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 0x7fdf840c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf89251700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840a7d00 0x7fdf84148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf88a50700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 0x7fdf840c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.99:57546/0 (socket says 172.21.15.99:57546) 2022-11-27T17:46:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38786/0 (socket says 172.21.15.99:38786) 2022-11-27T17:46:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf88a50700 1 -- 172.21.15.99:0/2027588669 learned_addr learned my addr 172.21.15.99:0/2027588669 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 -- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840a7d00 msgr2=0x7fdf84148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840a7d00 0x7fdf84148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 -- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 msgr2=0x7fdf840c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 0x7fdf840c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 -- 172.21.15.99:0/2027588669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf84147d00 con 0x7fdf840a7890 2022-11-27T17:46:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf83fff700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf70014000 tx=0x7fdf70001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70004c80 con 0x7fdf840a7890 2022-11-27T17:46:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70004de0 con 0x7fdf840a7890 2022-11-27T17:46:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf84147fe0 con 0x7fdf840a7890 2022-11-27T17:46:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700072d0 con 0x7fdf840a7890 2022-11-27T17:46:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.992+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf84149160 con 0x7fdf840a7890 2022-11-27T17:46:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.993+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf84148220 con 0x7fdf840a7890 2022-11-27T17:46:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.994+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fdf7002d070 con 0x7fdf840a7890 2022-11-27T17:46:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.994+0000 7fdf817fa700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 0x7fdf6c068a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.995+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5195+0+0 (secure 0 0 0) 0x7fdf70084670 con 0x7fdf840a7890 2022-11-27T17:46:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.995+0000 7fdf88a50700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 0x7fdf6c068a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.995+0000 7fdf88a50700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 0x7fdf6c068a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf78001870 tx=0x7fdf78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:00.998+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fdf70052230 con 0x7fdf840a7890 2022-11-27T17:46:01.182 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.181+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdf84148220 con 0x7fdf840a7890 2022-11-27T17:46:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.854+0000 7fdf817fa700 1 -- 172.21.15.99:0/2027588669 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7fdf84148220 con 0x7fdf840a7890 2022-11-27T17:46:01.859 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 msgr2=0x7fdf6c068a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 0x7fdf6c068a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf78001870 tx=0x7fdf78009040 comp rx=0 tx=0).stop 2022-11-27T17:46:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 msgr2=0x7fdf84148500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf70014000 tx=0x7fdf70001980 comp rx=0 tx=0).stop 2022-11-27T17:46:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 shutdown_connections 2022-11-27T17:46:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fdf840a7d00 0x7fdf84148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf68004850 0x7fdf840c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fdf6c0665c0 0x7fdf6c068a70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 --2- 172.21.15.99:0/2027588669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf840a7890 0x7fdf84148500 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 >> 172.21.15.99:0/2027588669 conn(0x7fdf84071060 msgr2=0x7fdf840b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 shutdown_connections 2022-11-27T17:46:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:01.858+0000 7fdf8acb4700 1 -- 172.21.15.99:0/2027588669 wait complete. 2022-11-27T17:46:01.863 INFO:teuthology.orchestra.run.smithi099.stderr:enabled application 'rbd' on pool 'rbd' 2022-11-27T17:46:01.875 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-11-27T17:46:01.876 INFO:tasks.ceph.mds.a:Restarting daemon 2022-11-27T17:46:01.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-11-27T17:46:01.879 INFO:tasks.ceph.mds.a:Started 2022-11-27T17:46:01.880 INFO:tasks.ceph.mds.c:Restarting daemon 2022-11-27T17:46:01.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-11-27T17:46:01.883 INFO:tasks.ceph.mds.c:Started 2022-11-27T17:46:01.883 INFO:tasks.ceph.mds.b:Restarting daemon 2022-11-27T17:46:01.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-11-27T17:46:01.886 INFO:tasks.ceph.mds.b:Started 2022-11-27T17:46:01.887 INFO:tasks.ceph.mds.d:Restarting daemon 2022-11-27T17:46:01.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-11-27T17:46:01.890 INFO:tasks.ceph.mds.d:Started 2022-11-27T17:46:01.891 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2022-11-27T17:46:01.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-27T17:46:01.966 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2022-11-27T17:46:01.972 INFO:tasks.ceph.mds.d.smithi119.stdout:starting mds.d at 2022-11-27T17:46:01.973 INFO:tasks.ceph.mds.b.smithi119.stdout:starting mds.b at 2022-11-27T17:46:01.974 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2022-11-27T17:46:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.032+0000 7fd0107b3700 1 Processor -- start 2022-11-27T17:46:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.032+0000 7fd0107b3700 1 -- start start 2022-11-27T17:46:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.033+0000 7fd0107b3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 0x7fd0080c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.033+0000 7fd0107b3700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.033+0000 7fd0107b3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0080c79e0 con 0x7fd0080650d0 2022-11-27T17:46:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.033+0000 7fd0107b3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0080c7c80 con 0x7fd00806ba90 2022-11-27T17:46:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.033+0000 7fd0107b3700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080c7f60 con 0x7fd00806c650 2022-11-27T17:46:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00e54f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 0x7fd0080c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00dd4e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd0080650d0 0x7fd00806be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.99:42374/0 (socket says 172.21.15.99:42374) 2022-11-27T17:46:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00dd4e700 1 -- 172.21.15.99:0/3865678478 learned_addr learned my addr 172.21.15.99:0/3865678478 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 --2- 172.21.15.99:0/3865678478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.99:43242/0 (socket says 172.21.15.99:43242) 2022-11-27T17:46:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 -- 172.21.15.99:0/3865678478 >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 msgr2=0x7fd0080c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 --2- 172.21.15.99:0/3865678478 >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 0x7fd0080c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 -- 172.21.15.99:0/3865678478 >> v1:172.21.15.119:6789/0 conn(0x7fd0080650d0 legacy=0x7fd00806be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00d54d700 1 -- 172.21.15.99:0/3865678478 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935639963 0 0) 0x7fd0080c79e0 con 0x7fd0080650d0 2022-11-27T17:46:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 -- 172.21.15.99:0/3865678478 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0080c8240 con 0x7fd00806c650 2022-11-27T17:46:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00e54f700 1 --2- 172.21.15.99:0/3865678478 >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 0x7fd0080c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-11-27T17:46:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00ed50700 1 --2- 172.21.15.99:0/3865678478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd000009dc0 tx=0x7fd00000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40b88b1aa4880e25 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00d54d700 1 -- 172.21.15.99:0/3865678478 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00000e070 con 0x7fd00806c650 2022-11-27T17:46:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00d54d700 1 -- 172.21.15.99:0/3865678478 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00000bae0 con 0x7fd00806c650 2022-11-27T17:46:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.034+0000 7fd00d54d700 1 -- 172.21.15.99:0/3865678478 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00000be90 con 0x7fd00806c650 2022-11-27T17:46:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 -- 172.21.15.99:0/3865678478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 msgr2=0x7fd00806f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3865678478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd000009dc0 tx=0x7fd00000b330 comp rx=0 tx=0).stop 2022-11-27T17:46:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 -- 172.21.15.99:0/3865678478 shutdown_connections 2022-11-27T17:46:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3865678478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd00806c650 0x7fd00806f9d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3865678478 >> v2:172.21.15.119:3300/0 conn(0x7fd00806ba90 0x7fd0080c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 -- 172.21.15.99:0/3865678478 >> 172.21.15.99:0/3865678478 conn(0x7fd0080b0f50 msgr2=0x7fd0080ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 -- 172.21.15.99:0/3865678478 shutdown_connections 2022-11-27T17:46:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.035+0000 7fd0107b3700 1 -- 172.21.15.99:0/3865678478 wait complete. 2022-11-27T17:46:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.036+0000 7fd0107b3700 1 Processor -- start 2022-11-27T17:46:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.036+0000 7fd0107b3700 1 -- start start 2022-11-27T17:46:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 0x7fd00814d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd00806ba90 0x7fd008140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 0x7fd0080b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00814d1f0 con 0x7fd00806c650 2022-11-27T17:46:02.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00814c5e0 con 0x7fd00806ba90 2022-11-27T17:46:02.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00814c8c0 con 0x7fd0080650d0 2022-11-27T17:46:02.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00ed50700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 0x7fd0080b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 0x7fd00814d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00ed50700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 0x7fd0080b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38826/0 (socket says 172.21.15.99:38826) 2022-11-27T17:46:02.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00ed50700 1 -- 172.21.15.99:0/3398088822 learned_addr learned my addr 172.21.15.99:0/3398088822 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:02.054 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00dd4e700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd00806ba90 0x7fd008140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.054 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 -- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd00806ba90 msgr2=0x7fd008140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.054 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd00806ba90 0x7fd008140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 -- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 msgr2=0x7fd0080b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 0x7fd0080b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 -- 172.21.15.99:0/3398088822 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80094a0 con 0x7fd0080650d0 2022-11-27T17:46:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd00e54f700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 0x7fd00814d680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcff800e900 tx=0x7fcff800ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8013680 con 0x7fd0080650d0 2022-11-27T17:46:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff80137e0 con 0x7fd0080650d0 2022-11-27T17:46:02.058 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd000009a70 con 0x7fd0080650d0 2022-11-27T17:46:02.058 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.037+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8013b70 con 0x7fd0080650d0 2022-11-27T17:46:02.059 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.038+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00814ce80 con 0x7fd0080650d0 2022-11-27T17:46:02.059 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.038+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd008073030 con 0x7fd0080650d0 2022-11-27T17:46:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.039+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fcff8019030 con 0x7fd0080650d0 2022-11-27T17:46:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.039+0000 7fcff6ffd700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 0x7fcfe8068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.039+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcff80832a0 con 0x7fd0080650d0 2022-11-27T17:46:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.039+0000 7fd00dd4e700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 0x7fcfe8068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.039+0000 7fd00dd4e700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 0x7fcfe8068a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0080ce1e0 tx=0x7fd004007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.062 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.044+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fcff8050d00 con 0x7fd0080650d0 2022-11-27T17:46:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.227+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd00806cc10 con 0x7fd0080650d0 2022-11-27T17:46:02.230 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.228+0000 7fcff6ffd700 1 -- 172.21.15.99:0/3398088822 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13754 (secure 0 0 0) 0x7fcff8054320 con 0x7fd0080650d0 2022-11-27T17:46:02.230 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:46:02.231 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":19,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:46:01.848852+0000","last_up_change":"2022-11-27T17:45:53.769684+0000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-27T17:45:55.780809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-11-27T17:46:00.824346+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":101360},{"type":"v1","addr":"172.21.15.99:6817","nonce":101360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":101360},{"type":"v1","addr":"172.21.15.99:6820","nonce":101360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":101360},{"type":"v1","addr":"172.21.15.99:6828","nonce":101360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":101360},{"type":"v1","addr":"172.21.15.99:6824","nonce":101360}]},"public_addr":"172.21.15.99:6817/101360","cluster_addr":"172.21.15.99:6820/101360","heartbeat_back_addr":"172.21.15.99:6828/101360","heartbeat_front_addr":"172.21.15.99:6824/101360","state":["exists","up"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":101359},{"type":"v1","addr":"172.21.15.99:6801","nonce":101359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":101359},{"type":"v1","addr":"172.21.15.99:6803","nonce":101359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":101359},{"type":"v1","addr":"172.21.15.99:6807","nonce":101359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":101359},{"type":"v1","addr":"172.21.15.99:6805","nonce":101359}]},"public_addr":"172.21.15.99:6801/101359","cluster_addr":"172.21.15.99:6803/101359","heartbeat_back_addr":"172.21.15.99:6807/101359","heartbeat_front_addr":"172.21.15.99:6805/101359","state":["exists","up"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":101358},{"type":"v1","addr":"172.21.15.99:6809","nonce":101358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":101358},{"type":"v1","addr":"172.21.15.99:6811","nonce":101358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":101358},{"type":"v1","addr":"172.21.15.99:6815","nonce":101358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":101358},{"type":"v1","addr":"172.21.15.99:6813","nonce":101358}]},"public_addr":"172.21.15.99:6809/101358","cluster_addr":"172.21.15.99:6811/101358","heartbeat_back_addr":"172.21.15.99:6815/101358","heartbeat_front_addr":"172.21.15.99:6813/101358","state":["exists","up"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":101361},{"type":"v1","addr":"172.21.15.99:6821","nonce":101361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6823","nonce":101361},{"type":"v1","addr":"172.21.15.99:6825","nonce":101361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":101361},{"type":"v1","addr":"172.21.15.99:6831","nonce":101361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6827","nonce":101361},{"type":"v1","addr":"172.21.15.99:6829","nonce":101361}]},"public_addr":"172.21.15.99:6821/101361","cluster_addr":"172.21.15.99:6825/101361","heartbeat_back_addr":"172.21.15.99:6831/101361","heartbeat_front_addr":"172.21.15.99:6829/101361","state":["exists","up"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":101614},{"type":"v1","addr":"172.21.15.119:6801","nonce":101614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":101614},{"type":"v1","addr":"172.21.15.119:6805","nonce":101614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":101614},{"type":"v1","addr":"172.21.15.119:6812","nonce":101614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":101614},{"type":"v1","addr":"172.21.15.119:6808","nonce":101614}]},"public_addr":"172.21.15.119:6801/101614","cluster_addr":"172.21.15.119:6805/101614","heartbeat_back_addr":"172.21.15.119:6812/101614","heartbeat_front_addr":"172.21.15.119:6808/101614","state":["exists","up"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":101615},{"type":"v1","addr":"172.21.15.119:6804","nonce":101615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":101615},{"type":"v1","addr":"172.21.15.119:6809","nonce":101615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":101615},{"type":"v1","addr":"172.21.15.119:6815","nonce":101615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":101615},{"type":"v1","addr":"172.21.15.119:6813","nonce":101615}]},"public_addr":"172.21.15.119:6804/101615","cluster_addr":"172.21.15.119:6809/101615","heartbeat_back_addr":"172.21.15.119:6815/101615","heartbeat_front_addr":"172.21.15.119:6813/101615","state":["exists","up"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":101644},{"type":"v1","addr":"172.21.15.119:6817","nonce":101644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":101644},{"type":"v1","addr":"172.21.15.119:6819","nonce":101644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":101644},{"type":"v1","addr":"172.21.15.119:6823","nonce":101644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":101644},{"type":"v1","addr":"172.21.15.119:6821","nonce":101644}]},"public_addr":"172.21.15.119:6817/101644","cluster_addr":"172.21.15.119:6819/101644","heartbeat_back_addr":"172.21.15.119:6823/101644","heartbeat_front_addr":"172.21.15.119:6821/101644","state":["exists","up"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":101645},{"type":"v1","addr":"172.21.15.119:6825","nonce":101645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":101645},{"type":"v1","addr":"172.21.15.119:6827","nonce":101645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":101645},{"type":"v1","addr":"172.21.15.119:6831","nonce":101645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":101645},{"type":"v1","addr":"172.21.15.119:6829","nonce":101645}]},"public_addr":"172.21.15.119:6825/101645","cluster_addr":"172.21.15.119:6827/101645","heartbeat_back_addr":"172.21.15.119:6831/101645","heartbeat_front_addr":"172.21.15.119:6829/101645","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.255830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.201490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.145828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.190338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.159710+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.218916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.273223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:46:02.234 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 msgr2=0x7fcfe8068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 0x7fcfe8068a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0080ce1e0 tx=0x7fd004007b90 comp rx=0 tx=0).stop 2022-11-27T17:46:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 msgr2=0x7fd00814d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 0x7fd00814d680 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcff800e900 tx=0x7fcff800ec10 comp rx=0 tx=0).stop 2022-11-27T17:46:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 shutdown_connections 2022-11-27T17:46:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fcfe8066590 0x7fcfe8068a40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd00806c650 0x7fd0080b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd0080650d0 0x7fd00814d680 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 --2- 172.21.15.99:0/3398088822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd00806ba90 0x7fd008140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 >> 172.21.15.99:0/3398088822 conn(0x7fd0080b0f50 msgr2=0x7fd0081541e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 shutdown_connections 2022-11-27T17:46:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.231+0000 7fd0107b3700 1 -- 172.21.15.99:0/3398088822 wait complete. 2022-11-27T17:46:02.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-11-27T17:45:55.780809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-11-27T17:46:00.824346+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-11-27T17:46:02.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-11-27T17:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.368+0000 7f754b561700 1 Processor -- start 2022-11-27T17:46:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f754b561700 1 -- start start 2022-11-27T17:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f754b561700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75440b0760 0x7f75440693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f754b561700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440ce050 con 0x7f75440ccd20 2022-11-27T17:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f754b561700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440cc240 con 0x7f754415ba40 2022-11-27T17:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f754b561700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440cc520 con 0x7f75440b0760 2022-11-27T17:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f7548afc700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f75440ccd20 0x7f75440b0050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53086/0 (socket says 172.21.15.99:53086) 2022-11-27T17:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.369+0000 7f7548afc700 1 -- 172.21.15.99:0/2049619797 learned_addr learned my addr 172.21.15.99:0/2049619797 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7549afe700 1 --2- 172.21.15.99:0/2049619797 >> v2:172.21.15.119:3300/0 conn(0x7f75440b0760 0x7f75440693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998372458 0 0) 0x7f75440ce050 con 0x7f75440ccd20 2022-11-27T17:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0035b0 con 0x7f75440ccd20 2022-11-27T17:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423227854 0 0) 0x7f75440cc240 con 0x7f754415ba40 2022-11-27T17:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75440ce050 con 0x7f754415ba40 2022-11-27T17:46:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7549afe700 1 -- 172.21.15.99:0/2049619797 >> v1:172.21.15.119:6789/0 conn(0x7f754415ba40 legacy=0x7f75440cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7549afe700 1 -- 172.21.15.99:0/2049619797 >> v1:172.21.15.99:6789/0 conn(0x7f75440ccd20 legacy=0x7f75440b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7549afe700 1 -- 172.21.15.99:0/2049619797 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440cc800 con 0x7f75440b0760 2022-11-27T17:46:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7549afe700 1 --2- 172.21.15.99:0/2049619797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75440b0760 0x7f75440693f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f753c009dc0 tx=0x7f753c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe4f427011ec9d04 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00e070 con 0x7f75440b0760 2022-11-27T17:46:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c00bae0 con 0x7f75440b0760 2022-11-27T17:46:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.370+0000 7f7533fff700 1 -- 172.21.15.99:0/2049619797 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00be90 con 0x7f75440b0760 2022-11-27T17:46:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 -- 172.21.15.99:0/2049619797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75440b0760 msgr2=0x7f75440693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 --2- 172.21.15.99:0/2049619797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75440b0760 0x7f75440693f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f753c009dc0 tx=0x7f753c00b330 comp rx=0 tx=0).stop 2022-11-27T17:46:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 -- 172.21.15.99:0/2049619797 shutdown_connections 2022-11-27T17:46:02.380 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 --2- 172.21.15.99:0/2049619797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75440b0760 0x7f75440693f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.380 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 -- 172.21.15.99:0/2049619797 >> 172.21.15.99:0/2049619797 conn(0x7f75440cb050 msgr2=0x7f75440cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.381 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 -- 172.21.15.99:0/2049619797 shutdown_connections 2022-11-27T17:46:02.381 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.371+0000 7f754b561700 1 -- 172.21.15.99:0/2049619797 wait complete. 2022-11-27T17:46:02.382 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.372+0000 7f754b561700 1 Processor -- start 2022-11-27T17:46:02.382 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.372+0000 7f754b561700 1 -- start start 2022-11-27T17:46:02.389 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 0x7f75440c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.389 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.389 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 0x7f75440b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.389 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75441e4b30 con 0x7f75440ccd20 2022-11-27T17:46:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75441e41d0 con 0x7f754415ba40 2022-11-27T17:46:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75441e4470 con 0x7f75440b0760 2022-11-27T17:46:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38838/0 (socket says 172.21.15.99:38838) 2022-11-27T17:46:02.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 -- 172.21.15.99:0/1991676428 learned_addr learned my addr 172.21.15.99:0/1991676428 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:02.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f75492fd700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 0x7f75440c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7549afe700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 0x7f75440b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 -- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 msgr2=0x7f75440c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 0x7f75440c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 -- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 msgr2=0x7f75440b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 0x7f75440b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 -- 172.21.15.99:0/1991676428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c009a70 con 0x7f75440ccd20 2022-11-27T17:46:02.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f75492fd700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 0x7f75440c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-11-27T17:46:02.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7549afe700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 0x7f75440b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-11-27T17:46:02.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7548afc700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f754000c700 tx=0x7f754000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540005d70 con 0x7f75440ccd20 2022-11-27T17:46:02.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75400070d0 con 0x7f75440ccd20 2022-11-27T17:46:02.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75441e4750 con 0x7f75440ccd20 2022-11-27T17:46:02.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.373+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540007480 con 0x7f75440ccd20 2022-11-27T17:46:02.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.374+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75440b2b70 con 0x7f75440ccd20 2022-11-27T17:46:02.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.375+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544073030 con 0x7f75440ccd20 2022-11-27T17:46:02.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.375+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f7540019030 con 0x7f75440ccd20 2022-11-27T17:46:02.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.375+0000 7f7531ffb700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 0x7f7524068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.376+0000 7f75492fd700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 0x7f7524068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.396 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.376+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f75400829c0 con 0x7f75440ccd20 2022-11-27T17:46:02.396 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.376+0000 7f75492fd700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 0x7f7524068a40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75340011e0 tx=0x7f7534009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.396 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.380+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f75400503a0 con 0x7f75440ccd20 2022-11-27T17:46:02.561 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.560+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75440ce330 con 0x7f75440ccd20 2022-11-27T17:46:02.561 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.560+0000 7f7531ffb700 1 -- 172.21.15.99:0/1991676428 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75400170a0 con 0x7f75440ccd20 2022-11-27T17:46:02.562 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2022-11-27T17:46:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 msgr2=0x7f7524068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 0x7f7524068a40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75340011e0 tx=0x7f7534009040 comp rx=0 tx=0).stop 2022-11-27T17:46:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 msgr2=0x7f75440c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f754000c700 tx=0x7f754000cac0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 shutdown_connections 2022-11-27T17:46:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f754415ba40 0x7f75440b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f75440b0760 0x7f75440c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f7524066590 0x7f7524068a40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 --2- 172.21.15.99:0/1991676428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75440ccd20 0x7f75440c56a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 >> 172.21.15.99:0/1991676428 conn(0x7f75440cb050 msgr2=0x7f75440693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.562+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 shutdown_connections 2022-11-27T17:46:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.563+0000 7f754b561700 1 -- 172.21.15.99:0/1991676428 wait complete. 2022-11-27T17:46:02.576 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-11-27T17:46:02.699 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.698+0000 7fb3c4cf6700 1 Processor -- start 2022-11-27T17:46:02.699 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.698+0000 7fb3c4cf6700 1 -- start start 2022-11-27T17:46:02.707 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3c006ff80 0x7fb3c0070350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 0x7fb3c00c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.708 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb3c00ce510 0x7fb3c006c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c0070890 con 0x7fb3c006ff80 2022-11-27T17:46:02.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c00c4b00 con 0x7fb3c00ce510 2022-11-27T17:46:02.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3c4cf6700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00cda30 con 0x7fb3c00c4620 2022-11-27T17:46:02.709 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3c006ff80 0x7fb3c0070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3c006ff80 0x7fb3c0070350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38844/0 (socket says 172.21.15.99:38844) 2022-11-27T17:46:02.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 -- 172.21.15.99:0/135943946 learned_addr learned my addr 172.21.15.99:0/135943946 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:02.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3bed9d700 1 --2- 172.21.15.99:0/135943946 >> v2:172.21.15.119:3300/0 conn(0x7fb3c00ce510 0x7fb3c006c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.710 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3bdd9b700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 0x7fb3c00c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 -- 172.21.15.99:0/135943946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 msgr2=0x7fb3c00c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 0x7fb3c00c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 -- 172.21.15.99:0/135943946 >> v2:172.21.15.119:3300/0 conn(0x7fb3c00ce510 msgr2=0x7fb3c006c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 --2- 172.21.15.99:0/135943946 >> v2:172.21.15.119:3300/0 conn(0x7fb3c00ce510 0x7fb3c006c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 -- 172.21.15.99:0/135943946 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c00cdd10 con 0x7fb3c006ff80 2022-11-27T17:46:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3bdd9b700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 0x7fb3c00c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3be59c700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c006ff80 0x7fb3c0070350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3c006b2f0 tx=0x7fb3b000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2259c13cdbd47785 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.699+0000 7fb3bd59a700 1 -- 172.21.15.99:0/135943946 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b000dc70 con 0x7fb3c006ff80 2022-11-27T17:46:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3bd59a700 1 -- 172.21.15.99:0/135943946 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b000ddd0 con 0x7fb3c006ff80 2022-11-27T17:46:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3bd59a700 1 -- 172.21.15.99:0/135943946 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b00106a0 con 0x7fb3c006ff80 2022-11-27T17:46:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/135943946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c006ff80 msgr2=0x7fb3c0070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c006ff80 0x7fb3c0070350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3c006b2f0 tx=0x7fb3b000be70 comp rx=0 tx=0).stop 2022-11-27T17:46:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/135943946 shutdown_connections 2022-11-27T17:46:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/135943946 >> v2:172.21.15.119:3300/0 conn(0x7fb3c00ce510 0x7fb3c006c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c006ff80 0x7fb3c0070350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.715 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/135943946 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00c4620 0x7fb3c00c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.715 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/135943946 >> 172.21.15.99:0/135943946 conn(0x7fb3c00b16c0 msgr2=0x7fb3c00cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.715 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/135943946 shutdown_connections 2022-11-27T17:46:02.716 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.700+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/135943946 wait complete. 2022-11-27T17:46:02.716 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 Processor -- start 2022-11-27T17:46:02.717 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 -- start start 2022-11-27T17:46:02.717 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3c006ff80 0x7fb3c014b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.717 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.717 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00ce510 0x7fb3c01ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.718 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c00b3d90 con 0x7fb3c00c4620 2022-11-27T17:46:02.718 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01ed9c0 con 0x7fb3c006ff80 2022-11-27T17:46:02.718 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3c4cf6700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01edca0 con 0x7fb3c00ce510 2022-11-27T17:46:02.718 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3bdd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.719 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3bdd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38852/0 (socket says 172.21.15.99:38852) 2022-11-27T17:46:02.719 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3bdd9b700 1 -- 172.21.15.99:0/2925278340 learned_addr learned my addr 172.21.15.99:0/2925278340 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:02.719 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3be59c700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3c006ff80 0x7fb3c014b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.702+0000 7fb3bed9d700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00ce510 0x7fb3c01ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 -- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00ce510 msgr2=0x7fb3c01ecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00ce510 0x7fb3c01ecf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 -- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3c006ff80 msgr2=0x7fb3c014b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3c006ff80 0x7fb3c014b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 -- 172.21.15.99:0/2925278340 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b000b7f0 con 0x7fb3c00c4620 2022-11-27T17:46:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3bdd9b700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400bac0 tx=0x7fb3b400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4004230 con 0x7fb3c00c4620 2022-11-27T17:46:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b4013040 con 0x7fb3c00c4620 2022-11-27T17:46:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c00b39b0 con 0x7fb3c00c4620 2022-11-27T17:46:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4004590 con 0x7fb3c00c4620 2022-11-27T17:46:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.703+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c01f06b0 con 0x7fb3c00c4620 2022-11-27T17:46:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.704+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c00b4000 con 0x7fb3c00c4620 2022-11-27T17:46:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.704+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fb3b401e030 con 0x7fb3c00c4620 2022-11-27T17:46:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.705+0000 7fb3aeffd700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 0x7fb39c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:02.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.705+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb3b4086820 con 0x7fb3c00c4620 2022-11-27T17:46:02.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.705+0000 7fb3be59c700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 0x7fb39c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:02.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.705+0000 7fb3be59c700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 0x7fb39c068a40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0008010 tx=0x7fb3b001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:02.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.709+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fb3b404fbd0 con 0x7fb3c00c4620 2022-11-27T17:46:02.892 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.890+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb3c00b3810 con 0x7fb3c00c4620 2022-11-27T17:46:02.892 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.891+0000 7fb3aeffd700 1 -- 172.21.15.99:0/2925278340 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb3c00b3810 con 0x7fb3c00c4620 2022-11-27T17:46:02.892 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 8 2022-11-27T17:46:02.894 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 msgr2=0x7fb39c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.894 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 0x7fb39c068a40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0008010 tx=0x7fb3b001c040 comp rx=0 tx=0).stop 2022-11-27T17:46:02.895 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 msgr2=0x7fb3c00b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:02.895 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400bac0 tx=0x7fb3b400bdd0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.896 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 shutdown_connections 2022-11-27T17:46:02.896 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb3c00ce510 0x7fb3c01ecf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.897 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3c006ff80 0x7fb3c014b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.897 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fb39c066590 0x7fb39c068a40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.897 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 --2- 172.21.15.99:0/2925278340 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3c00c4620 0x7fb3c00b41b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:02.898 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 >> 172.21.15.99:0/2925278340 conn(0x7fb3c00b16c0 msgr2=0x7fb3c014e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:02.898 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 shutdown_connections 2022-11-27T17:46:02.899 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:02.893+0000 7fb3c4cf6700 1 -- 172.21.15.99:0/2925278340 wait complete. 2022-11-27T17:46:02.906 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-11-27T17:46:03.026 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.025+0000 7fca54980700 1 Processor -- start 2022-11-27T17:46:03.026 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.025+0000 7fca54980700 1 -- start start 2022-11-27T17:46:03.029 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 0x7fca500cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.030 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fca500cc9a0 0x7fca500cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.030 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca500cff20 0x7fca500d0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.030 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500cfba0 con 0x7fca500cc9a0 2022-11-27T17:46:03.030 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500d08a0 con 0x7fca500cff20 2022-11-27T17:46:03.031 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca54980700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015f610 con 0x7fca5015f240 2022-11-27T17:46:03.031 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fca500cc9a0 0x7fca500cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.031 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fca500cc9a0 0x7fca500cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38862/0 (socket says 172.21.15.99:38862) 2022-11-27T17:46:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2390251963 learned_addr learned my addr 172.21.15.99:0/2390251963 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4ed9d700 1 --2- 172.21.15.99:0/2390251963 >> v2:172.21.15.119:3300/0 conn(0x7fca500cff20 0x7fca500d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4e59c700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 0x7fca500cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2390251963 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 msgr2=0x7fca500cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 0x7fca500cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2390251963 >> v2:172.21.15.119:3300/0 conn(0x7fca500cff20 msgr2=0x7fca500d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2390251963 >> v2:172.21.15.119:3300/0 conn(0x7fca500cff20 0x7fca500d0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2390251963 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500cdfc0 con 0x7fca500cc9a0 2022-11-27T17:46:03.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4e59c700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 0x7fca500cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cc9a0 0x7fca500cf060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca5008c720 tx=0x7fca44009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf3dc31682ed069 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.026+0000 7fca4d59a700 1 -- 172.21.15.99:0/2390251963 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44011830 con 0x7fca500cc9a0 2022-11-27T17:46:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca4d59a700 1 -- 172.21.15.99:0/2390251963 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca44011990 con 0x7fca500cc9a0 2022-11-27T17:46:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca4d59a700 1 -- 172.21.15.99:0/2390251963 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44011d20 con 0x7fca500cc9a0 2022-11-27T17:46:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 -- 172.21.15.99:0/2390251963 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cc9a0 msgr2=0x7fca500cf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cc9a0 0x7fca500cf060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca5008c720 tx=0x7fca44009bd0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 -- 172.21.15.99:0/2390251963 shutdown_connections 2022-11-27T17:46:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 --2- 172.21.15.99:0/2390251963 >> v2:172.21.15.119:3300/0 conn(0x7fca500cff20 0x7fca500d0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca5015f240 0x7fca500cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 --2- 172.21.15.99:0/2390251963 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cc9a0 0x7fca500cf060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 -- 172.21.15.99:0/2390251963 >> 172.21.15.99:0/2390251963 conn(0x7fca5009b560 msgr2=0x7fca500d2350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 -- 172.21.15.99:0/2390251963 shutdown_connections 2022-11-27T17:46:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.027+0000 7fca54980700 1 -- 172.21.15.99:0/2390251963 wait complete. 2022-11-27T17:46:03.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.028+0000 7fca54980700 1 Processor -- start 2022-11-27T17:46:03.039 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 -- start start 2022-11-27T17:46:03.044 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca500cc9a0 0x7fca501f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 0x7fca5014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5014bfa0 con 0x7fca500cff20 2022-11-27T17:46:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501f4370 con 0x7fca5015f240 2022-11-27T17:46:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca54980700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca500b7600 con 0x7fca500cc9a0 2022-11-27T17:46:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca4dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca4dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38868/0 (socket says 172.21.15.99:38868) 2022-11-27T17:46:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.029+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2927255748 learned_addr learned my addr 172.21.15.99:0/2927255748 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4ed9d700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 0x7fca5014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4e59c700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca500cc9a0 0x7fca501f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca500cc9a0 msgr2=0x7fca501f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca500cc9a0 0x7fca501f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 msgr2=0x7fca5014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 0x7fca5014e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 -- 172.21.15.99:0/2927255748 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca44009550 con 0x7fca500cff20 2022-11-27T17:46:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4ed9d700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 0x7fca5014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca4dd9b700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca44009c10 tx=0x7fca44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca440036a0 con 0x7fca500cff20 2022-11-27T17:46:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4401e070 con 0x7fca500cff20 2022-11-27T17:46:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca500b78e0 con 0x7fca500cff20 2022-11-27T17:46:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44003a60 con 0x7fca500cff20 2022-11-27T17:46:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.030+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca500b7ea0 con 0x7fca500cff20 2022-11-27T17:46:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.031+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50073030 con 0x7fca500cff20 2022-11-27T17:46:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.031+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fca4400f050 con 0x7fca500cff20 2022-11-27T17:46:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.032+0000 7fca3effd700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 0x7fca2c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.032+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fca44084300 con 0x7fca500cff20 2022-11-27T17:46:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.032+0000 7fca4e59c700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 0x7fca2c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.032+0000 7fca4e59c700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 0x7fca2c068a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca40009930 tx=0x7fca40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.036+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fca44051d60 con 0x7fca500cff20 2022-11-27T17:46:03.261 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.260+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca500cda40 con 0x7fca500cff20 2022-11-27T17:46:03.262 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.261+0000 7fca3effd700 1 -- 172.21.15.99:0/2927255748 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3530 (secure 0 0 0) 0x7fca44020de0 con 0x7fca500cff20 2022-11-27T17:46:03.263 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:46:03.264 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/1144417062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":1144417062},{"type":"v1","addr":"172.21.15.99:6835","nonce":1144417062}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6834/1061854086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1061854086},{"type":"v1","addr":"172.21.15.119:6834","nonce":1061854086}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6837/3013599057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6836","nonce":3013599057},{"type":"v1","addr":"172.21.15.99:6837","nonce":3013599057}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1019820821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6833","nonce":1019820821},{"type":"v1","addr":"172.21.15.119:6835","nonce":1019820821}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-11-27T17:46:03.266 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 msgr2=0x7fca2c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.267 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 0x7fca2c068a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca40009930 tx=0x7fca40008040 comp rx=0 tx=0).stop 2022-11-27T17:46:03.267 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 msgr2=0x7fca5014e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.267 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca44009c10 tx=0x7fca44004750 comp rx=0 tx=0).stop 2022-11-27T17:46:03.268 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 shutdown_connections 2022-11-27T17:46:03.268 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5015f240 0x7fca5014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.269 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fca500cc9a0 0x7fca501f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.269 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fca2c066590 0x7fca2c068a40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.270 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 --2- 172.21.15.99:0/2927255748 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca500cff20 0x7fca5014e000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.271 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 >> 172.21.15.99:0/2927255748 conn(0x7fca5009b560 msgr2=0x7fca500b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.271 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 shutdown_connections 2022-11-27T17:46:03.272 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.263+0000 7fca54980700 1 -- 172.21.15.99:0/2927255748 wait complete. 2022-11-27T17:46:03.272 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 3 2022-11-27T17:46:03.277 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-11-27T17:46:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.396+0000 7f13c1fd5700 1 Processor -- start 2022-11-27T17:46:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 -- start start 2022-11-27T17:46:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc15f1b0 0x7f13bc0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f13bc06ac10 0x7f13bc0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0d08a0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13bc06c5a0 con 0x7f13bc06ac10 2022-11-27T17:46:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13c1fd5700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc15f580 con 0x7f13bc15f1b0 2022-11-27T17:46:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13bbfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f13bc06ac10 0x7f13bc0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13bbfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f13bc06ac10 0x7f13bc0ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38878/0 (socket says 172.21.15.99:38878) 2022-11-27T17:46:03.410 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.397+0000 7f13bbfff700 1 -- 172.21.15.99:0/225628070 learned_addr learned my addr 172.21.15.99:0/225628070 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.410 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13bbfff700 1 -- 172.21.15.99:0/225628070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc15f1b0 msgr2=0x7f13bc0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:03.410 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13bbfff700 1 --2- 172.21.15.99:0/225628070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc15f1b0 0x7f13bc0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.411 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13bbfff700 1 -- 172.21.15.99:0/225628070 >> v1:172.21.15.119:6789/0 conn(0x7f13bc0cf7f0 legacy=0x7f13bc06a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:03.411 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13bbfff700 1 -- 172.21.15.99:0/225628070 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0cfc90 con 0x7f13bc06ac10 2022-11-27T17:46:03.412 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13bbfff700 1 --2- 172.21.15.99:0/225628070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc06ac10 0x7f13bc0ca6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009e50 tx=0x7f13b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797d3bd7b3d4994d server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.412 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13ba7fc700 1 -- 172.21.15.99:0/225628070 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400e070 con 0x7f13bc06ac10 2022-11-27T17:46:03.413 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13ba7fc700 1 -- 172.21.15.99:0/225628070 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4003a20 con 0x7f13bc06ac10 2022-11-27T17:46:03.413 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13ba7fc700 1 -- 172.21.15.99:0/225628070 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4003dd0 con 0x7f13bc06ac10 2022-11-27T17:46:03.413 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13c1fd5700 1 -- 172.21.15.99:0/225628070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc06ac10 msgr2=0x7f13bc0ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.414 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.398+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/225628070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc06ac10 0x7f13bc0ca6c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13b4009e50 tx=0x7f13b400b330 comp rx=0 tx=0).stop 2022-11-27T17:46:03.414 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 -- 172.21.15.99:0/225628070 shutdown_connections 2022-11-27T17:46:03.415 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/225628070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc06ac10 0x7f13bc0ca6c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.415 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/225628070 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc15f1b0 0x7f13bc0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.416 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 -- 172.21.15.99:0/225628070 >> 172.21.15.99:0/225628070 conn(0x7f13bc069d10 msgr2=0x7f13bc099350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.416 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 -- 172.21.15.99:0/225628070 shutdown_connections 2022-11-27T17:46:03.416 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.399+0000 7f13c1fd5700 1 -- 172.21.15.99:0/225628070 wait complete. 2022-11-27T17:46:03.417 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.400+0000 7f13c1fd5700 1 Processor -- start 2022-11-27T17:46:03.417 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.400+0000 7f13c1fd5700 1 -- start start 2022-11-27T17:46:03.418 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc06ac10 0x7f13bc1e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.418 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.419 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13bc15f1b0 0x7f13bc153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.419 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1f23a0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.419 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1e48f0 con 0x7f13bc15f1b0 2022-11-27T17:46:03.420 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc1e4bd0 con 0x7f13bc06ac10 2022-11-27T17:46:03.420 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.421 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38882/0 (socket says 172.21.15.99:38882) 2022-11-27T17:46:03.421 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 -- 172.21.15.99:0/2088514368 learned_addr learned my addr 172.21.15.99:0/2088514368 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.422 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13bbfff700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13bc15f1b0 0x7f13bc153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.422 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13bb7fe700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc06ac10 0x7f13bc1e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.422 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 -- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc06ac10 msgr2=0x7f13bc1e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.423 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc06ac10 0x7f13bc1e5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.423 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 -- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13bc15f1b0 msgr2=0x7f13bc153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.423 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13bc15f1b0 0x7f13bc153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.424 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 -- 172.21.15.99:0/2088514368 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4009b30 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.424 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13baffd700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c4a0 tx=0x7f13a400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.425 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40054e0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.425 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a4003680 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.426 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4003a10 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.426 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc1e4eb0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.426 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.401+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc1e5470 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.427 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.402+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc073030 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.427 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.403+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f13a4010070 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.428 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.403+0000 7f13b3fff700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 0x7f13a0068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.428 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.403+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f13a40822c0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.403+0000 7f13bb7fe700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 0x7f13a0068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.404+0000 7f13bb7fe700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 0x7f13a0068a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13ac006fd0 tx=0x7f13ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.430 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.407+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f13a4049bd0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.586+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13bc0cff70 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.587+0000 7f13b3fff700 1 -- 172.21.15.99:0/2088514368 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13741 (secure 0 0 0) 0x7f13a404fdd0 con 0x7f13bc0cf7f0 2022-11-27T17:46:03.589 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:46:03.589 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":20,"fsid":"03183fc1-5cf9-40d3-b24e-1fc3808ae993","created":"2022-11-27T17:45:42.460043+0000","modified":"2022-11-27T17:46:02.853053+0000","last_up_change":"2022-11-27T17:45:53.769684+0000","last_in_change":"2022-11-27T17:45:44.931976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-11-27T17:45:55.780809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-11-27T17:46:00.824346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a82a87a1-4c37-42e5-b3c8-42f429d47bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":101360},{"type":"v1","addr":"172.21.15.99:6817","nonce":101360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":101360},{"type":"v1","addr":"172.21.15.99:6820","nonce":101360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":101360},{"type":"v1","addr":"172.21.15.99:6828","nonce":101360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":101360},{"type":"v1","addr":"172.21.15.99:6824","nonce":101360}]},"public_addr":"172.21.15.99:6817/101360","cluster_addr":"172.21.15.99:6820/101360","heartbeat_back_addr":"172.21.15.99:6828/101360","heartbeat_front_addr":"172.21.15.99:6824/101360","state":["exists","up"]},{"osd":1,"uuid":"a688c5d6-7c05-4bba-8590-c2029d25c345","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":101359},{"type":"v1","addr":"172.21.15.99:6801","nonce":101359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":101359},{"type":"v1","addr":"172.21.15.99:6803","nonce":101359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":101359},{"type":"v1","addr":"172.21.15.99:6807","nonce":101359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":101359},{"type":"v1","addr":"172.21.15.99:6805","nonce":101359}]},"public_addr":"172.21.15.99:6801/101359","cluster_addr":"172.21.15.99:6803/101359","heartbeat_back_addr":"172.21.15.99:6807/101359","heartbeat_front_addr":"172.21.15.99:6805/101359","state":["exists","up"]},{"osd":2,"uuid":"04ae94e5-c567-40e0-b979-63a599c1f2e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":101358},{"type":"v1","addr":"172.21.15.99:6809","nonce":101358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":101358},{"type":"v1","addr":"172.21.15.99:6811","nonce":101358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":101358},{"type":"v1","addr":"172.21.15.99:6815","nonce":101358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":101358},{"type":"v1","addr":"172.21.15.99:6813","nonce":101358}]},"public_addr":"172.21.15.99:6809/101358","cluster_addr":"172.21.15.99:6811/101358","heartbeat_back_addr":"172.21.15.99:6815/101358","heartbeat_front_addr":"172.21.15.99:6813/101358","state":["exists","up"]},{"osd":3,"uuid":"f2da7a20-eb09-4b7a-9f10-5e2178b3cb9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":101361},{"type":"v1","addr":"172.21.15.99:6821","nonce":101361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6823","nonce":101361},{"type":"v1","addr":"172.21.15.99:6825","nonce":101361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":101361},{"type":"v1","addr":"172.21.15.99:6831","nonce":101361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6827","nonce":101361},{"type":"v1","addr":"172.21.15.99:6829","nonce":101361}]},"public_addr":"172.21.15.99:6821/101361","cluster_addr":"172.21.15.99:6825/101361","heartbeat_back_addr":"172.21.15.99:6831/101361","heartbeat_front_addr":"172.21.15.99:6829/101361","state":["exists","up"]},{"osd":4,"uuid":"f36aa763-c864-431b-a1f8-f7a5d0253246","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":101614},{"type":"v1","addr":"172.21.15.119:6801","nonce":101614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":101614},{"type":"v1","addr":"172.21.15.119:6805","nonce":101614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":101614},{"type":"v1","addr":"172.21.15.119:6812","nonce":101614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":101614},{"type":"v1","addr":"172.21.15.119:6808","nonce":101614}]},"public_addr":"172.21.15.119:6801/101614","cluster_addr":"172.21.15.119:6805/101614","heartbeat_back_addr":"172.21.15.119:6812/101614","heartbeat_front_addr":"172.21.15.119:6808/101614","state":["exists","up"]},{"osd":5,"uuid":"cafb58cd-ca0e-4dae-bf69-9ac81df4074e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":101615},{"type":"v1","addr":"172.21.15.119:6804","nonce":101615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":101615},{"type":"v1","addr":"172.21.15.119:6809","nonce":101615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":101615},{"type":"v1","addr":"172.21.15.119:6815","nonce":101615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":101615},{"type":"v1","addr":"172.21.15.119:6813","nonce":101615}]},"public_addr":"172.21.15.119:6804/101615","cluster_addr":"172.21.15.119:6809/101615","heartbeat_back_addr":"172.21.15.119:6815/101615","heartbeat_front_addr":"172.21.15.119:6813/101615","state":["exists","up"]},{"osd":6,"uuid":"46754ecc-48dc-417a-be57-c27f20466fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":101644},{"type":"v1","addr":"172.21.15.119:6817","nonce":101644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":101644},{"type":"v1","addr":"172.21.15.119:6819","nonce":101644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":101644},{"type":"v1","addr":"172.21.15.119:6823","nonce":101644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":101644},{"type":"v1","addr":"172.21.15.119:6821","nonce":101644}]},"public_addr":"172.21.15.119:6817/101644","cluster_addr":"172.21.15.119:6819/101644","heartbeat_back_addr":"172.21.15.119:6823/101644","heartbeat_front_addr":"172.21.15.119:6821/101644","state":["exists","up"]},{"osd":7,"uuid":"48b889c7-f370-4749-bb57-a26ebf4eec3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":101645},{"type":"v1","addr":"172.21.15.119:6825","nonce":101645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":101645},{"type":"v1","addr":"172.21.15.119:6827","nonce":101645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":101645},{"type":"v1","addr":"172.21.15.119:6831","nonce":101645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":101645},{"type":"v1","addr":"172.21.15.119:6829","nonce":101645}]},"public_addr":"172.21.15.119:6825/101645","cluster_addr":"172.21.15.119:6827/101645","heartbeat_back_addr":"172.21.15.119:6831/101645","heartbeat_front_addr":"172.21.15.119:6829/101645","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.255830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.108856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.201490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.145828+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.190338+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.159710+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.218916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2022-11-27T17:45:50.273223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-11-27T17:46:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 msgr2=0x7f13a0068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 0x7f13a0068a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13ac006fd0 tx=0x7f13ac008040 comp rx=0 tx=0).stop 2022-11-27T17:46:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 msgr2=0x7f13bc1f1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f13a400c4a0 tx=0x7f13a400c7b0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 shutdown_connections 2022-11-27T17:46:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13bc15f1b0 0x7f13bc153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f13bc06ac10 0x7f13bc1e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f13a0066590 0x7f13a0068a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 --2- 172.21.15.99:0/2088514368 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f13bc0cf7f0 0x7f13bc1f1e60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 >> 172.21.15.99:0/2088514368 conn(0x7f13bc069d10 msgr2=0x7f13bc0b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 shutdown_connections 2022-11-27T17:46:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.589+0000 7f13c1fd5700 1 -- 172.21.15.99:0/2088514368 wait complete. 2022-11-27T17:46:03.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-11-27T17:45:55.780809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-11-27T17:46:00.824346+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-11-27T17:46:03.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-11-27T17:46:03.728 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.726+0000 7fea9bdb2700 1 Processor -- start 2022-11-27T17:46:03.728 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 -- start start 2022-11-27T17:46:03.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 0x7fea940c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea940b1a80 0x7fea9406bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9406af10 con 0x7fea9406c180 2022-11-27T17:46:03.740 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940c36d0 con 0x7fea940b1a80 2022-11-27T17:46:03.740 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9bdb2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9406b160 con 0x7fea9415fe70 2022-11-27T17:46:03.741 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.727+0000 7fea9934d700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea940b1a80 0x7fea9406bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.741 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea940b1a80 0x7fea9406bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38890/0 (socket says 172.21.15.99:38890) 2022-11-27T17:46:03.741 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 -- 172.21.15.99:0/1562513106 learned_addr learned my addr 172.21.15.99:0/1562513106 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.742 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea99b4e700 1 --2- 172.21.15.99:0/1562513106 >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 0x7fea940c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.742 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 -- 172.21.15.99:0/1562513106 >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 msgr2=0x7fea940c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:03.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 --2- 172.21.15.99:0/1562513106 >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 0x7fea940c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 -- 172.21.15.99:0/1562513106 >> v1:172.21.15.119:6789/0 conn(0x7fea9406c180 legacy=0x7fea9406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.743 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 -- 172.21.15.99:0/1562513106 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9406b440 con 0x7fea940b1a80 2022-11-27T17:46:03.744 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea99b4e700 1 --2- 172.21.15.99:0/1562513106 >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 0x7fea940c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:03.744 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea9934d700 1 --2- 172.21.15.99:0/1562513106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea940b1a80 0x7fea9406bc40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea940cd0b0 tx=0x7fea90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae2f5a68952f8b3 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.745 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea98b4c700 1 -- 172.21.15.99:0/1562513106 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9000b870 con 0x7fea940b1a80 2022-11-27T17:46:03.745 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea98b4c700 1 -- 172.21.15.99:0/1562513106 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9000b9d0 con 0x7fea940b1a80 2022-11-27T17:46:03.745 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.728+0000 7fea98b4c700 1 -- 172.21.15.99:0/1562513106 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9000bd60 con 0x7fea940b1a80 2022-11-27T17:46:03.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 -- 172.21.15.99:0/1562513106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea940b1a80 msgr2=0x7fea9406bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/1562513106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea940b1a80 0x7fea9406bc40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea940cd0b0 tx=0x7fea90009bd0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.746 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 -- 172.21.15.99:0/1562513106 shutdown_connections 2022-11-27T17:46:03.747 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/1562513106 >> v2:172.21.15.119:3300/0 conn(0x7fea9415fe70 0x7fea940c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.747 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/1562513106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea940b1a80 0x7fea9406bc40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.748 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 -- 172.21.15.99:0/1562513106 >> 172.21.15.99:0/1562513106 conn(0x7fea94071060 msgr2=0x7fea94071430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.748 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 -- 172.21.15.99:0/1562513106 shutdown_connections 2022-11-27T17:46:03.748 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.729+0000 7fea9bdb2700 1 -- 172.21.15.99:0/1562513106 wait complete. 2022-11-27T17:46:03.749 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.730+0000 7fea9bdb2700 1 Processor -- start 2022-11-27T17:46:03.749 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.730+0000 7fea9bdb2700 1 -- start start 2022-11-27T17:46:03.749 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fea9406c180 0x7fea941e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.750 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea940b1a80 0x7fea940c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.750 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.750 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea941e6750 con 0x7fea9415fe70 2022-11-27T17:46:03.751 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9415f640 con 0x7fea940b1a80 2022-11-27T17:46:03.751 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b2f40 con 0x7fea9406c180 2022-11-27T17:46:03.752 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.752 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38898/0 (socket says 172.21.15.99:38898) 2022-11-27T17:46:03.752 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 -- 172.21.15.99:0/723785496 learned_addr learned my addr 172.21.15.99:0/723785496 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:03.753 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9934d700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea940b1a80 0x7fea940c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.753 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 -- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fea9406c180 msgr2=0x7fea941e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.754 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea99b4e700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fea9406c180 0x7fea941e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.754 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fea9406c180 0x7fea941e6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.755 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 -- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea940b1a80 msgr2=0x7fea940c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.755 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea940b1a80 0x7fea940c4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.756 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 -- 172.21.15.99:0/723785496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea90009550 con 0x7fea9415fe70 2022-11-27T17:46:03.756 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9a34f700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00cc50 tx=0x7fea8c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.756 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00ad80 con 0x7fea9415fe70 2022-11-27T17:46:03.757 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c00ce50 con 0x7fea9415fe70 2022-11-27T17:46:03.757 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea941e56b0 con 0x7fea9415fe70 2022-11-27T17:46:03.758 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.731+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8c003c40 con 0x7fea9415fe70 2022-11-27T17:46:03.758 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.732+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea941e5c70 con 0x7fea9415fe70 2022-11-27T17:46:03.758 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.732+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea94073030 con 0x7fea9415fe70 2022-11-27T17:46:03.759 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.733+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7fea8c010050 con 0x7fea9415fe70 2022-11-27T17:46:03.759 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.733+0000 7fea827fc700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 0x7fea74068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:03.760 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.733+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fea8c0833d0 con 0x7fea9415fe70 2022-11-27T17:46:03.760 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.733+0000 7fea99b4e700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 0x7fea74068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:03.760 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.733+0000 7fea99b4e700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 0x7fea74068a40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea84009750 tx=0x7fea84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:03.760 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.737+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7fea8c050e30 con 0x7fea9415fe70 2022-11-27T17:46:03.918 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.916+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea9406b720 con 0x7fea9415fe70 2022-11-27T17:46:03.919 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.917+0000 7fea827fc700 1 -- 172.21.15.99:0/723785496 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea8c054450 con 0x7fea9415fe70 2022-11-27T17:46:03.919 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2022-11-27T17:46:03.922 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 msgr2=0x7fea74068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.922 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 0x7fea74068a40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea84009750 tx=0x7fea84008040 comp rx=0 tx=0).stop 2022-11-27T17:46:03.922 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 msgr2=0x7fea940c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:03.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00cc50 tx=0x7fea8c00a340 comp rx=0 tx=0).stop 2022-11-27T17:46:03.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 shutdown_connections 2022-11-27T17:46:03.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7fea74066590 0x7fea74068a40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea9415fe70 0x7fea940c51f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.923 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fea9406c180 0x7fea941e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.924 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 --2- 172.21.15.99:0/723785496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea940b1a80 0x7fea940c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:03.924 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 >> 172.21.15.99:0/723785496 conn(0x7fea94071060 msgr2=0x7fea9415db20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:03.924 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 shutdown_connections 2022-11-27T17:46:03.924 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:03.919+0000 7fea9bdb2700 1 -- 172.21.15.99:0/723785496 wait complete. 2022-11-27T17:46:03.932 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-11-27T17:46:04.055 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.054+0000 7f0160c5c700 1 Processor -- start 2022-11-27T17:46:04.056 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.054+0000 7f0160c5c700 1 -- start start 2022-11-27T17:46:04.060 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f0160c5c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f015c0cf8c0 0x7f015c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f0160c5c700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c06c050 con 0x7f015c15bb10 2022-11-27T17:46:04.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f0160c5c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c069c40 con 0x7f015c15bee0 2022-11-27T17:46:04.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f0160c5c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c069f20 con 0x7f015c0cf8c0 2022-11-27T17:46:04.061 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f015a59c700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f015c15bb10 0x7f015c0693f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53156/0 (socket says 172.21.15.99:53156) 2022-11-27T17:46:04.062 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f015a59c700 1 -- 172.21.15.99:0/2616589494 learned_addr learned my addr 172.21.15.99:0/2616589494 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:04.062 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.055+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713308634 0 0) 0x7f015c06c050 con 0x7f015c15bb10 2022-11-27T17:46:04.062 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01400035b0 con 0x7f015c15bb10 2022-11-27T17:46:04.063 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 --2- 172.21.15.99:0/2616589494 >> v2:172.21.15.119:3300/0 conn(0x7f015c0cf8c0 0x7f015c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.063 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942123013 0 0) 0x7f01400035b0 con 0x7f015c15bb10 2022-11-27T17:46:04.063 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 >> v2:172.21.15.119:3300/0 conn(0x7f015c0cf8c0 msgr2=0x7f015c06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:04.063 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 --2- 172.21.15.99:0/2616589494 >> v2:172.21.15.119:3300/0 conn(0x7f015c0cf8c0 0x7f015c06d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.064 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 >> v1:172.21.15.119:6789/0 conn(0x7f015c15bee0 legacy=0x7f015c06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.064 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c06a200 con 0x7f015c15bb10 2022-11-27T17:46:04.064 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2447476370 0 0) 0x7f014c001670 con 0x7f015c15bb10 2022-11-27T17:46:04.064 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 >> v1:172.21.15.99:6789/0 conn(0x7f015c15bb10 legacy=0x7f015c0693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.065 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0140003a00 0x7f0140003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.065 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0140004850 0x7f0140004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.066 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f014000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.066 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c069f20 con 0x7f0140005200 2022-11-27T17:46:04.066 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014000c190 con 0x7f0140003a00 2022-11-27T17:46:04.066 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014000c350 con 0x7f0140004850 2022-11-27T17:46:04.067 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f014000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.067 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015a59c700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0140003a00 0x7f0140003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.067 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f0159d9b700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0140004850 0x7f0140004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 -- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0140004850 msgr2=0x7f0140004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0140004850 0x7f0140004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 -- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0140003a00 msgr2=0x7f0140003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0140003a00 0x7f0140003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.068 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 -- 172.21.15.99:0/2616589494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01400035b0 con 0x7f0140005200 2022-11-27T17:46:04.069 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 -- 172.21.15.99:0/2616589494 reap_dead start 2022-11-27T17:46:04.069 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015ad9d700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f014000ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0154007330 tx=0x7f0140005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff43a0ece4e9a37b server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.069 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154007e50 con 0x7f0140005200 2022-11-27T17:46:04.069 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0154003a20 con 0x7f0140005200 2022-11-27T17:46:04.070 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.056+0000 7f015959a700 1 -- 172.21.15.99:0/2616589494 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154003dd0 con 0x7f0140005200 2022-11-27T17:46:04.070 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 -- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 msgr2=0x7f014000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.070 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f014000ba10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0154007330 tx=0x7f0140005880 comp rx=0 tx=0).stop 2022-11-27T17:46:04.070 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 -- 172.21.15.99:0/2616589494 shutdown_connections 2022-11-27T17:46:04.071 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 --2- 172.21.15.99:0/2616589494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f014000ba10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.071 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 -- 172.21.15.99:0/2616589494 >> 172.21.15.99:0/2616589494 conn(0x7f015c0b16c0 msgr2=0x7f015c0cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:04.071 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 -- 172.21.15.99:0/2616589494 shutdown_connections 2022-11-27T17:46:04.071 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.057+0000 7f0160c5c700 1 -- 172.21.15.99:0/2616589494 wait complete. 2022-11-27T17:46:04.072 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.059+0000 7f0160c5c700 1 Processor -- start 2022-11-27T17:46:04.072 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.059+0000 7f0160c5c700 1 -- start start 2022-11-27T17:46:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 0x7f015c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f015c15cfc0 0x7f015c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c157890 con 0x7f0140005200 2022-11-27T17:46:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c145160 con 0x7f015c156ee0 2022-11-27T17:46:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c144760 con 0x7f015c15cfc0 2022-11-27T17:46:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38914/0 (socket says 172.21.15.99:38914) 2022-11-27T17:46:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 -- 172.21.15.99:0/809111082 learned_addr learned my addr 172.21.15.99:0/809111082 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0159d9b700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 0x7f015c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 -- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f015c15cfc0 msgr2=0x7f015c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015ad9d700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f015c15cfc0 0x7f015c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f015c15cfc0 0x7f015c15d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 -- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 msgr2=0x7f015c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 0x7f015c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 -- 172.21.15.99:0/809111082 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c15dfb0 con 0x7f0140005200 2022-11-27T17:46:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0159d9b700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 0x7f015c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-11-27T17:46:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f015a59c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f014c0072c0 tx=0x7f014c0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00f6c0 con 0x7f0140005200 2022-11-27T17:46:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00f820 con 0x7f0140005200 2022-11-27T17:46:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c15dcd0 con 0x7f0140005200 2022-11-27T17:46:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.060+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00fbd0 con 0x7f0140005200 2022-11-27T17:46:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.061+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c144df0 con 0x7f0140005200 2022-11-27T17:46:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.061+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c065ac0 con 0x7f0140005200 2022-11-27T17:46:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.062+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f014c00fd30 con 0x7f0140005200 2022-11-27T17:46:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.062+0000 7f014affd700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 0x7f0138068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.062+0000 7f0159d9b700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 0x7f0138068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.062+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f014c0834e0 con 0x7f0140005200 2022-11-27T17:46:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.063+0000 7f0159d9b700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 0x7f0138068a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f015c1f4e20 tx=0x7f0150009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.066+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f014c050ee0 con 0x7f0140005200 2022-11-27T17:46:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.247+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f015c144f80 con 0x7f0140005200 2022-11-27T17:46:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.247+0000 7f014affd700 1 -- 172.21.15.99:0/809111082 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f015c144f80 con 0x7f0140005200 2022-11-27T17:46:04.249 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 8 2022-11-27T17:46:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 msgr2=0x7f0138068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 0x7f0138068a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f015c1f4e20 tx=0x7f0150009040 comp rx=0 tx=0).stop 2022-11-27T17:46:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 msgr2=0x7f015c144220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f014c0072c0 tx=0x7f014c0075d0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 shutdown_connections 2022-11-27T17:46:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f0138066590 0x7f0138068a40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0140005200 0x7f015c144220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f015c15cfc0 0x7f015c15d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.254 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 --2- 172.21.15.99:0/809111082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f015c156ee0 0x7f015c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.254 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 >> 172.21.15.99:0/809111082 conn(0x7f015c0b16c0 msgr2=0x7f015c0b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:04.255 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 shutdown_connections 2022-11-27T17:46:04.255 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.249+0000 7f0160c5c700 1 -- 172.21.15.99:0/809111082 wait complete. 2022-11-27T17:46:04.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-11-27T17:46:04.386 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.385+0000 7f5ab2c32700 1 Processor -- start 2022-11-27T17:46:04.386 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.385+0000 7f5ab2c32700 1 -- start start 2022-11-27T17:46:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab2c32700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5aac0650d0 0x7f5aac0c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab2c32700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06c650 0x7f5aac06f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab2c32700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aac0c79e0 con 0x7f5aac06ba90 2022-11-27T17:46:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab2c32700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aac0c7c80 con 0x7f5aac0650d0 2022-11-27T17:46:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab2c32700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0c7f60 con 0x7f5aac06c650 2022-11-27T17:46:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5aac0650d0 0x7f5aac0c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5aac0650d0 0x7f5aac0c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38924/0 (socket says 172.21.15.99:38924) 2022-11-27T17:46:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 -- 172.21.15.99:0/3474314468 learned_addr learned my addr 172.21.15.99:0/3474314468 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5ab11cf700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06c650 0x7f5aac06f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 -- 172.21.15.99:0/3474314468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06c650 msgr2=0x7f5aac06f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06c650 0x7f5aac06f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 -- 172.21.15.99:0/3474314468 >> v1:172.21.15.119:6789/0 conn(0x7f5aac06ba90 legacy=0x7f5aac06be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 -- 172.21.15.99:0/3474314468 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac0c8240 con 0x7f5aac0650d0 2022-11-27T17:46:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aabfff700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0c8c40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009920 tx=0x7f5a98009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fab6ede6156577 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aab7fe700 1 -- 172.21.15.99:0/3474314468 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800ba20 con 0x7f5aac0650d0 2022-11-27T17:46:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aab7fe700 1 -- 172.21.15.99:0/3474314468 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9800bb80 con 0x7f5aac0650d0 2022-11-27T17:46:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.386+0000 7f5aab7fe700 1 -- 172.21.15.99:0/3474314468 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98009e70 con 0x7f5aac0650d0 2022-11-27T17:46:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3474314468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 msgr2=0x7f5aac0c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0c8c40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009920 tx=0x7f5a98009c30 comp rx=0 tx=0).stop 2022-11-27T17:46:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3474314468 shutdown_connections 2022-11-27T17:46:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06c650 0x7f5aac06f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3474314468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0c8c40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3474314468 >> 172.21.15.99:0/3474314468 conn(0x7f5aac0b0f50 msgr2=0x7f5aac0ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3474314468 shutdown_connections 2022-11-27T17:46:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.387+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3474314468 wait complete. 2022-11-27T17:46:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.388+0000 7f5ab2c32700 1 Processor -- start 2022-11-27T17:46:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.388+0000 7f5ab2c32700 1 -- start start 2022-11-27T17:46:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06ba90 0x7f5aac149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aac06c650 0x7f5aac0cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0cbc70 con 0x7f5aac0650d0 2022-11-27T17:46:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac1e6fc0 con 0x7f5aac06c650 2022-11-27T17:46:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab2c32700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac1e72a0 con 0x7f5aac06ba90 2022-11-27T17:46:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38928/0 (socket says 172.21.15.99:38928) 2022-11-27T17:46:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 -- 172.21.15.99:0/3721996095 learned_addr learned my addr 172.21.15.99:0/3721996095 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab11cf700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aac06c650 0x7f5aac0cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5aabfff700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06ba90 0x7f5aac149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 -- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06ba90 msgr2=0x7f5aac149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06ba90 0x7f5aac149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 -- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aac06c650 msgr2=0x7f5aac0cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.389+0000 7f5ab09ce700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aac06c650 0x7f5aac0cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5ab09ce700 1 -- 172.21.15.99:0/3721996095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98009550 con 0x7f5aac0650d0 2022-11-27T17:46:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5ab09ce700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c450 tx=0x7f5aa000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00052d0 con 0x7f5aac0650d0 2022-11-27T17:46:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa0005430 con 0x7f5aac0650d0 2022-11-27T17:46:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac0caf30 con 0x7f5aac0650d0 2022-11-27T17:46:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00038c0 con 0x7f5aac0650d0 2022-11-27T17:46:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.390+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac14a520 con 0x7f5aac0650d0 2022-11-27T17:46:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.391+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac073030 con 0x7f5aac0650d0 2022-11-27T17:46:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.391+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f5aa0019030 con 0x7f5aac0650d0 2022-11-27T17:46:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.391+0000 7f5aa97fa700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 0x7f5a94068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.391+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5aa00826a0 con 0x7f5aac0650d0 2022-11-27T17:46:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.391+0000 7f5aabfff700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 0x7f5a94068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.392+0000 7f5aabfff700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 0x7f5a94068a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a98000c00 tx=0x7f5a98008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.406 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.395+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f5aa0050080 con 0x7f5aac0650d0 2022-11-27T17:46:04.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.575+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5aac06cc10 con 0x7f5aac0650d0 2022-11-27T17:46:04.577 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.576+0000 7f5aa97fa700 1 -- 172.21.15.99:0/3721996095 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+126 (secure 0 0 0) 0x7f5aa0017090 con 0x7f5aac0650d0 2022-11-27T17:46:04.578 INFO:teuthology.orchestra.run.smithi099.stdout: 2022-11-27T17:46:04.579 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2022-11-27T17:46:04.580 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-11-27T17:46:04.580 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2022-11-27T17:46:04.580 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2022-11-27T17:46:04.581 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2022-11-27T17:46:04.581 INFO:teuthology.orchestra.run.smithi099.stdout: { 2022-11-27T17:46:04.582 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 2, 2022-11-27T17:46:04.582 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": "rbd" 2022-11-27T17:46:04.583 INFO:teuthology.orchestra.run.smithi099.stdout: } 2022-11-27T17:46:04.585 INFO:teuthology.orchestra.run.smithi099.stdout:] 2022-11-27T17:46:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 msgr2=0x7f5a94068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 0x7f5a94068a40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a98000c00 tx=0x7f5a98008030 comp rx=0 tx=0).stop 2022-11-27T17:46:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 msgr2=0x7f5aac0cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5aa000c450 tx=0x7f5aa000c760 comp rx=0 tx=0).stop 2022-11-27T17:46:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 shutdown_connections 2022-11-27T17:46:04.589 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f5a94066590 0x7f5a94068a40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.589 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5aac0650d0 0x7f5aac0cdab0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.590 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aac06c650 0x7f5aac0cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.590 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 --2- 172.21.15.99:0/3721996095 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aac06ba90 0x7f5aac149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.591 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 >> 172.21.15.99:0/3721996095 conn(0x7f5aac0b0f50 msgr2=0x7f5aac0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:04.591 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 shutdown_connections 2022-11-27T17:46:04.592 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.577+0000 7f5ab2c32700 1 -- 172.21.15.99:0/3721996095 wait complete. 2022-11-27T17:46:04.594 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-11-27T17:46:04.594 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-11-27T17:46:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.717+0000 7f6084f74700 1 Processor -- start 2022-11-27T17:46:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.717+0000 7f6084f74700 1 -- start start 2022-11-27T17:46:04.721 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f6084f74700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f608006d830 0x7f60800b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f6084f74700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800b1ca0 con 0x7f60801598c0 2022-11-27T17:46:04.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f6084f74700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800d0cb0 con 0x7f60800b2dc0 2022-11-27T17:46:04.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f6084f74700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60800cc830 con 0x7f608006d830 2022-11-27T17:46:04.722 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f607ed9d700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f608006d830 0x7f60800b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f607e59c700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f60801598c0 0x7f60800d05a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53190/0 (socket says 172.21.15.99:53190) 2022-11-27T17:46:04.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f607ed9d700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f608006d830 0x7f60800b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38940/0 (socket says 172.21.15.99:38940) 2022-11-27T17:46:04.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.718+0000 7f607e59c700 1 -- 172.21.15.99:0/2658712129 learned_addr learned my addr 172.21.15.99:0/2658712129 (peer_addr_for_me v1:172.21.15.99:0/0) 2022-11-27T17:46:04.723 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607ed9d700 1 -- 172.21.15.99:0/2658712129 >> v1:172.21.15.119:6789/0 conn(0x7f60800b2dc0 legacy=0x7f608006d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:04.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607ed9d700 1 -- 172.21.15.99:0/2658712129 >> v1:172.21.15.99:6789/0 conn(0x7f60801598c0 legacy=0x7f60800d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607ed9d700 1 -- 172.21.15.99:0/2658712129 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800cc270 con 0x7f608006d830 2022-11-27T17:46:04.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607d59a700 1 -- 172.21.15.99:0/2658712129 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566196380 0 0) 0x7f60800b1ca0 con 0x7f60801598c0 2022-11-27T17:46:04.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607ed9d700 1 --2- 172.21.15.99:0/2658712129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f608006d830 0x7f60800b3230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6078009d90 tx=0x7f607800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a1df0c15f0027f server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.724 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607d59a700 1 -- 172.21.15.99:0/2658712129 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607800e070 con 0x7f608006d830 2022-11-27T17:46:04.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607d59a700 1 -- 172.21.15.99:0/2658712129 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607800bb90 con 0x7f608006d830 2022-11-27T17:46:04.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.719+0000 7f607d59a700 1 -- 172.21.15.99:0/2658712129 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078003a20 con 0x7f608006d830 2022-11-27T17:46:04.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 -- 172.21.15.99:0/2658712129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f608006d830 msgr2=0x7f60800b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.725 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 --2- 172.21.15.99:0/2658712129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f608006d830 0x7f60800b3230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6078009d90 tx=0x7f607800b330 comp rx=0 tx=0).stop 2022-11-27T17:46:04.726 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 -- 172.21.15.99:0/2658712129 shutdown_connections 2022-11-27T17:46:04.726 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 --2- 172.21.15.99:0/2658712129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f608006d830 0x7f60800b3230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.726 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 -- 172.21.15.99:0/2658712129 >> 172.21.15.99:0/2658712129 conn(0x7f6080071060 msgr2=0x7f6080071430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:04.726 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 -- 172.21.15.99:0/2658712129 shutdown_connections 2022-11-27T17:46:04.727 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.720+0000 7f6084f74700 1 -- 172.21.15.99:0/2658712129 wait complete. 2022-11-27T17:46:04.730 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.721+0000 7f6084f74700 1 Processor -- start 2022-11-27T17:46:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.721+0000 7f6084f74700 1 -- start start 2022-11-27T17:46:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f608006d830 0x7f60800c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 0x7f60800c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.732 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60801561d0 con 0x7f60800b2dc0 2022-11-27T17:46:04.732 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60800c8b50 con 0x7f608006d830 2022-11-27T17:46:04.732 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60800c7c60 con 0x7f60801598c0 2022-11-27T17:46:04.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38944/0 (socket says 172.21.15.99:38944) 2022-11-27T17:46:04.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 -- 172.21.15.99:0/3318354442 learned_addr learned my addr 172.21.15.99:0/3318354442 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:04.733 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607e59c700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f608006d830 0x7f60800c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607ed9d700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 0x7f60800c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 -- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 msgr2=0x7f60800c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 0x7f60800c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 -- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f608006d830 msgr2=0x7f60800c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f608006d830 0x7f60800c66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:04.735 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 -- 172.21.15.99:0/3318354442 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078009a70 con 0x7f60800b2dc0 2022-11-27T17:46:04.735 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607ed9d700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 0x7f60800c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:04.735 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f607dd9b700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f607000c4d0 tx=0x7f607000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.735 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070011790 con 0x7f60800b2dc0 2022-11-27T17:46:04.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60700118f0 con 0x7f60800b2dc0 2022-11-27T17:46:04.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070011ca0 con 0x7f60800b2dc0 2022-11-27T17:46:04.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60800c7f40 con 0x7f60800b2dc0 2022-11-27T17:46:04.736 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.722+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60800c8500 con 0x7f60800b2dc0 2022-11-27T17:46:04.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.723+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60800cffe0 con 0x7f60800b2dc0 2022-11-27T17:46:04.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.724+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f6070010050 con 0x7f60800b2dc0 2022-11-27T17:46:04.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.724+0000 7f606effd700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 0x7f605c068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:04.737 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.724+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f60700830e0 con 0x7f60800b2dc0 2022-11-27T17:46:04.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.724+0000 7f607e59c700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 0x7f605c068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:04.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.724+0000 7f607e59c700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 0x7f605c068a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6074000f60 tx=0x7f6074007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:04.738 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.728+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f6070050b40 con 0x7f60800b2dc0 2022-11-27T17:46:04.911 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:04.909+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f60800c8cc0 con 0x7f60800b2dc0 2022-11-27T17:46:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.874+0000 7f606effd700 1 -- 172.21.15.99:0/3318354442 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v21) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6070054160 con 0x7f60800b2dc0 2022-11-27T17:46:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 msgr2=0x7f605c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 0x7f605c068a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6074000f60 tx=0x7f6074007e70 comp rx=0 tx=0).stop 2022-11-27T17:46:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 msgr2=0x7f6080154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f607000c4d0 tx=0x7f607000c7e0 comp rx=0 tx=0).stop 2022-11-27T17:46:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 shutdown_connections 2022-11-27T17:46:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60801598c0 0x7f60800c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f608006d830 0x7f60800c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f605c066590 0x7f605c068a40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 --2- 172.21.15.99:0/3318354442 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60800b2dc0 0x7f6080154ff0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 >> 172.21.15.99:0/3318354442 conn(0x7f6080071060 msgr2=0x7f6080157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.879+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 shutdown_connections 2022-11-27T17:46:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:05.880+0000 7f6084f74700 1 -- 172.21.15.99:0/3318354442 wait complete. 2022-11-27T17:46:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2022-11-27T17:46:05.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-11-27T17:46:06.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-11-27T17:46:06.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-11-27T17:46:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 Processor -- start 2022-11-27T17:46:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 -- start start 2022-11-27T17:46:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc15bb10 0x7f66fc0693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f66fc15bee0 0x7f66fc06c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc06c050 con 0x7f66fc06cd70 2022-11-27T17:46:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc069c40 con 0x7f66fc15bee0 2022-11-27T17:46:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.031+0000 7f6702af2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc069f20 con 0x7f66fc15bb10 2022-11-27T17:46:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f66fc15bee0 0x7f66fc06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f670088e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc15bb10 0x7f66fc0693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f66fc15bee0 0x7f66fc06c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38952/0 (socket says 172.21.15.99:38952) 2022-11-27T17:46:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f670108f700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f66fc06cd70 0x7f66fc06d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53208/0 (socket says 172.21.15.99:53208) 2022-11-27T17:46:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 -- 172.21.15.99:0/3235166904 learned_addr learned my addr 172.21.15.99:0/3235166904 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 -- 172.21.15.99:0/3235166904 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc15bb10 msgr2=0x7f66fc0693f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 --2- 172.21.15.99:0/3235166904 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc15bb10 0x7f66fc0693f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 -- 172.21.15.99:0/3235166904 >> v1:172.21.15.99:6789/0 conn(0x7f66fc06cd70 legacy=0x7f66fc06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 -- 172.21.15.99:0/3235166904 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc06a200 con 0x7f66fc15bee0 2022-11-27T17:46:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fbfff700 1 --2- 172.21.15.99:0/3235166904 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc06c830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098f0 tx=0x7f66f0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718ade3361da5219 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fb7fe700 1 -- 172.21.15.99:0/3235166904 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0011830 con 0x7f66fc15bee0 2022-11-27T17:46:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fb7fe700 1 -- 172.21.15.99:0/3235166904 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0011990 con 0x7f66fc15bee0 2022-11-27T17:46:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.032+0000 7f66fb7fe700 1 -- 172.21.15.99:0/3235166904 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0011d20 con 0x7f66fc15bee0 2022-11-27T17:46:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 -- 172.21.15.99:0/3235166904 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 msgr2=0x7f66fc06c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 --2- 172.21.15.99:0/3235166904 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc06c830 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098f0 tx=0x7f66f0009c00 comp rx=0 tx=0).stop 2022-11-27T17:46:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 -- 172.21.15.99:0/3235166904 shutdown_connections 2022-11-27T17:46:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 --2- 172.21.15.99:0/3235166904 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc15bb10 0x7f66fc0693f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 --2- 172.21.15.99:0/3235166904 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc06c830 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 -- 172.21.15.99:0/3235166904 >> 172.21.15.99:0/3235166904 conn(0x7f66fc0b16c0 msgr2=0x7f66fc0cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 -- 172.21.15.99:0/3235166904 shutdown_connections 2022-11-27T17:46:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.033+0000 7f6702af2700 1 -- 172.21.15.99:0/3235166904 wait complete. 2022-11-27T17:46:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.034+0000 7f6702af2700 1 Processor -- start 2022-11-27T17:46:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.034+0000 7f6702af2700 1 -- start start 2022-11-27T17:46:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 0x7f66fc144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 0x7f66fc15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc148ef0 con 0x7f66fc15bee0 2022-11-27T17:46:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1e54c0 con 0x7f66fc15bb10 2022-11-27T17:46:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f6702af2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc1e51e0 con 0x7f66fc06cd70 2022-11-27T17:46:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38960/0 (socket says 172.21.15.99:38960) 2022-11-27T17:46:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 -- 172.21.15.99:0/1207725207 learned_addr learned my addr 172.21.15.99:0/1207725207 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f66fbfff700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 0x7f66fc15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670088e700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 0x7f66fc144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 -- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 msgr2=0x7f66fc144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 0x7f66fc144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 -- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 msgr2=0x7f66fc15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 0x7f66fc15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670108f700 1 -- 172.21.15.99:0/1207725207 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0009550 con 0x7f66fc15bee0 2022-11-27T17:46:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f66fbfff700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 0x7f66fc15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-11-27T17:46:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.035+0000 7f670088e700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 0x7f66fc144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-11-27T17:46:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f670108f700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66f400cba0 tx=0x7f66f400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f400ab80 con 0x7f66fc15bee0 2022-11-27T17:46:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f400ace0 con 0x7f66fc15bee0 2022-11-27T17:46:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc1488a0 con 0x7f66fc15bee0 2022-11-27T17:46:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4007270 con 0x7f66fc15bee0 2022-11-27T17:46:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.036+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc144ec0 con 0x7f66fc15bee0 2022-11-27T17:46:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.037+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc065ac0 con 0x7f66fc15bee0 2022-11-27T17:46:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.038+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 84497+0+0 (secure 0 0 0) 0x7f66f4019030 con 0x7f66fc15bee0 2022-11-27T17:46:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.038+0000 7f66f97fa700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 0x7f66dc068a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.038+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f66f4083770 con 0x7f66fc15bee0 2022-11-27T17:46:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.038+0000 7f670088e700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 0x7f66dc068a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.039+0000 7f670088e700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 0x7f66dc068a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009b60 tx=0x7f66ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.042+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+170936 (secure 0 0 0) 0x7f66f40510a0 con 0x7f66fc15bee0 2022-11-27T17:46:06.227 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.226+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f66fc0c4700 con 0x7f66fc15bee0 2022-11-27T17:46:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.874+0000 7f66f97fa700 1 -- 172.21.15.99:0/1207725207 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v22) v1 ==== 170+0+0 (secure 0 0 0) 0x7f66f40546c0 con 0x7f66fc15bee0 2022-11-27T17:46:06.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.881+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 msgr2=0x7f66dc068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.881+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 0x7f66dc068a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009b60 tx=0x7f66ec008040 comp rx=0 tx=0).stop 2022-11-27T17:46:06.883 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.881+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 msgr2=0x7f66fc15e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.881+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66f400cba0 tx=0x7f66f400cf60 comp rx=0 tx=0).stop 2022-11-27T17:46:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 shutdown_connections 2022-11-27T17:46:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:6832/101086,v1:172.21.15.99:6833/101086] conn(0x7f66dc066590 0x7f66dc068a40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66fc15bee0 0x7f66fc15e1f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f66fc06cd70 0x7f66fc144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 --2- 172.21.15.99:0/1207725207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66fc15bb10 0x7f66fc15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.882+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 >> 172.21.15.99:0/1207725207 conn(0x7f66fc0b16c0 msgr2=0x7f66fc15a440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-11-27T17:46:06.888 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.883+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 shutdown_connections 2022-11-27T17:46:06.888 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:06.883+0000 7f6702af2700 1 -- 172.21.15.99:0/1207725207 wait complete. 2022-11-27T17:46:06.892 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2022-11-27T17:46:06.904 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-11-27T17:46:07.029 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.028+0000 7f077a93d700 1 Processor -- start 2022-11-27T17:46:07.029 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.028+0000 7f077a93d700 1 -- start start 2022-11-27T17:46:07.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f077a93d700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f077415bb10 0x7f07740693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:07.032 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f077a93d700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f077415bee0 0x7f077406c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-11-27T17:46:07.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f077a93d700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077406c050 con 0x7f077406cd70 2022-11-27T17:46:07.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f077a93d700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0774069c40 con 0x7f077415bb10 2022-11-27T17:46:07.033 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f077a93d700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0774069f20 con 0x7f077415bee0 2022-11-27T17:46:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f077415bb10 0x7f07740693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f077415bb10 0x7f07740693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38964/0 (socket says 172.21.15.99:38964) 2022-11-27T17:46:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 -- 172.21.15.99:0/1892355239 learned_addr learned my addr 172.21.15.99:0/1892355239 (peer_addr_for_me v2:172.21.15.99:0/0) 2022-11-27T17:46:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0778eda700 1 --1- 172.21.15.99:0/1892355239 >> v1:172.21.15.99:6789/0 conn(0x7f077406cd70 0x7f077406d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53222/0 (socket says 172.21.15.99:53222) 2022-11-27T17:46:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f07737fe700 1 --2- 172.21.15.99:0/1892355239 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f077415bee0 0x7f077406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-11-27T17:46:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 -- 172.21.15.99:0/1892355239 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f077415bee0 msgr2=0x7f077406c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-11-27T17:46:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 --2- 172.21.15.99:0/1892355239 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f077415bee0 0x7f077406c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-11-27T17:46:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 -- 172.21.15.99:0/1892355239 >> v1:172.21.15.99:6789/0 conn(0x7f077406cd70 legacy=0x7f077406d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 -- 172.21.15.99:0/1892355239 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077406a200 con 0x7f077415bb10 2022-11-27T17:46:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0772ffd700 1 -- 172.21.15.99:0/1892355239 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474919017 0 0) 0x7f077406c050 con 0x7f077406cd70 2022-11-27T17:46:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0773fff700 1 --2- 172.21.15.99:0/1892355239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f077415bb10 0x7f07740693f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f07640098f0 tx=0x7f0764009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f2201b166d7a0d server_cookie=0 in_seq=0 out_seq=0 2022-11-27T17:46:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0772ffd700 1 -- 172.21.15.99:0/1892355239 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400b8a0 con 0x7f077415bb10 2022-11-27T17:46:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0772ffd700 1 -- 172.21.15.99:0/1892355239 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076400ba00 con 0x7f077415bb10 2022-11-27T17:46:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.029+0000 7f0772ffd700 1 -- 172.21.15.99:0/1892355239 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076400bd90 con 0x7f077415bb10 2022-11-27T17:46:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.030+0000 7f077a93d700 1 -- 172.21.15.99:0/1892355239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f077415bb10 msgr2=0x7f07740693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-11-27T17:46:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.030+0000 7f077a93d700 1 --2- 172.21.15.99:0/1892355239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f077415bb10 0x7f07740693f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f07640098f0 tx=0x7f0764009c00 comp rx=0 tx=0).stop 2022-11-27T17:46:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.030+0000 7f077a93d700 1 -- 172.21.15.99:0/1892355239 shutdown_connections 2022-11-27T17:46:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2022-11-27T17:46:07.030+0000 7f077a93d700 1 --2- 172.21.15.99:0/1892355239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f077415bb10 0x7f07740693f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1