2024-04-26T04:10:01.502 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T04:10:01.502 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530 --name yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi --description rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} -- /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/orig.config.yaml 2024-04-26T04:10:01.532 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T04:10:01.661 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530 branch: wip-yuri2-testing-2024-04-24-0914-squid description: rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} email: yweinste@redhat.com first_in_suite: false job_id: '7674530' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri2-testing-2024-04-24-0914-squid ceph: conf: client: debug client: 20 debug mgrc: 20 debug ms: 1 global: mon election default strategy: 1 mds: debug mgrc: 20 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug mgrc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: ddeaa91797339a19172c3036ff48cca58c12f448 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ddeaa91797339a19172c3036ff48cca58c12f448 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-04-24-0914-squid sha1: ddeaa91797339a19172c3036ff48cca58c12f448 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - mon.a - mgr.a - client.0 - - host.b - client.1 seed: 1848 sha1: ddeaa91797339a19172c3036ff48cca58c12f448 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-04-24-0914-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ddeaa91797339a19172c3036ff48cca58c12f448 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA19QJu9tRoUX9UlbOntWO0tv02f2K/Fm2bg7NRZt1R0uJ8xt4JKE6c8XbnphlmwbhQ2/oPyZznlcxtcS1qmC2U= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKp4a3VjINqqXaKOYIbdLtD8d9Om8OH5J5nOOYvdQ2sGITObSENb1VqlE/B8l9oQ2LMFv7RTR8AP7UDFC2fvGos= tasks: - install: null - cephadm: null - workunit: clients: client.1: - cephadm/create_iscsi_disks.sh - workunit: clients: client.0: - cephadm/test_dashboard_e2e.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:23:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T04:10:01.662 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa; will attempt to use it 2024-04-26T04:10:01.662 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks 2024-04-26T04:10:01.662 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T04:10:01.664 INFO:teuthology.task.internal:Checking packages... 2024-04-26T04:10:01.694 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ddeaa91797339a19172c3036ff48cca58c12f448' 2024-04-26T04:10:01.694 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T04:10:01.695 INFO:teuthology.packaging:ref: None 2024-04-26T04:10:01.695 INFO:teuthology.packaging:tag: None 2024-04-26T04:10:01.695 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:10:01.695 INFO:teuthology.packaging:sha1: ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:10:01.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:10:01.849 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2457.gddeaa917 2024-04-26T04:10:01.851 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T04:10:01.858 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T04:10:01.858 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T04:10:01.865 INFO:teuthology.task.internal:Saving configuration 2024-04-26T04:10:01.885 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T04:10:01.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T04:10:01.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 04:04:56.609114', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA19QJu9tRoUX9UlbOntWO0tv02f2K/Fm2bg7NRZt1R0uJ8xt4JKE6c8XbnphlmwbhQ2/oPyZznlcxtcS1qmC2U='} 2024-04-26T04:10:01.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 04:04:56.610441', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKp4a3VjINqqXaKOYIbdLtD8d9Om8OH5J5nOOYvdQ2sGITObSENb1VqlE/B8l9oQ2LMFv7RTR8AP7UDFC2fvGos='} 2024-04-26T04:10:01.937 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T04:10:01.944 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2024-04-26T04:10:01.944 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-04-26T04:10:01.944 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T04:10:02.015 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f234ca2ef70>, signals=[15]) 2024-04-26T04:10:02.015 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T04:10:02.022 INFO:teuthology.task.internal:Opening connections... 2024-04-26T04:10:02.022 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-26T04:10:02.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:10:02.143 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-04-26T04:10:02.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:10:02.222 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T04:10:02.229 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-26T04:10:02.247 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-26T04:10:02.248 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-26T04:10:02.305 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-04-26T04:10:02.305 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-04-26T04:10:02.305 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-04-26T04:10:02.305 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-04-26T04:10:02.306 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-04-26T04:10:02.306 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-04-26T04:10:02.306 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T04:10:02.306 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T04:10:02.307 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T04:10:02.308 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-26T04:10:02.330 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-04-26T04:10:02.349 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-04-26T04:10:02.349 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-04-26T04:10:02.407 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-04-26T04:10:02.407 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-04-26T04:10:02.407 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-04-26T04:10:02.407 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-04-26T04:10:02.407 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T04:10:02.408 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T04:10:02.409 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-04-26T04:10:02.438 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T04:10:02.446 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T04:10:02.454 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T04:10:02.454 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-26T04:10:02.457 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-04-26T04:10:02.474 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T04:10:02.480 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T04:10:02.480 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-26T04:10:02.514 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-04-26T04:10:02.614 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T04:10:02.665 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T04:10:02.680 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T04:10:02.680 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T04:10:02.680 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T04:10:02.681 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T04:10:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:10:02.697 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-04-26T04:10:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:10:02.699 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-26T04:10:02.755 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-437.el9.x86_64 2024-04-26T04:10:02.756 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-437.el9.x86_64 2024-04-26T04:10:02.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-04-26T04:10:02.758 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-04-26T04:10:02.758 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-04-26T04:10:02.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-04-26T04:10:05.374 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-04-26T04:10:06.018 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-04-26T04:10:07.259 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-26T04:10:07.852 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-26T04:10:11.509 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-26T04:10:11.853 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-26T04:10:13.132 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-26T04:10:13.418 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-04-26T04:10:14.844 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T04:10:18.241 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 21 MB 00:04 2024-04-26T04:10:19.877 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-26T04:10:21.792 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-26T04:10:21.792 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-26T04:10:21.885 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:10:21.886 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:10:21.886 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:10:22.003 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2024-04-26T04:10:22.501 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Apr 2024 04:10:19 AM UTC. 2024-04-26T04:10:22.619 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:10:22.633 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:22.633 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-26T04:10:22.633 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:22.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:22.635 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-26T04:10:22.635 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-26T04:10:22.635 INFO:teuthology.orchestra.run.smithi098.stderr:Operation aborted. 2024-04-26T04:10:22.691 INFO:teuthology.orchestra.run.smithi098.stdout:Is this ok [y/N]: 2024-04-26T04:10:22.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2024-04-26T04:10:23.186 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:04 ago on Fri 26 Apr 2024 04:10:19 AM UTC. 2024-04-26T04:10:23.301 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:10:23.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-26T04:10:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-26T04:10:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:10:23.315 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-26T04:10:23.721 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64.rpm 936 kB/s | 255 kB 00:00 2024-04-26T04:10:23.721 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:10:23.722 INFO:teuthology.orchestra.run.smithi098.stdout:Total 608 kB/s | 255 kB 00:00 2024-04-26T04:10:23.743 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:10:23.926 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:10:23.926 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:10:23.931 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:10:23.931 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:10:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:10:24.148 INFO:teuthology.orchestra.run.smithi098.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T04:10:24.148 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:24.891 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:24.891 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T04:10:25.196 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-26T04:10:25.196 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-26T04:10:25.290 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:10:25.291 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:10:25.291 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:10:25.413 DEBUG:teuthology.orchestra.run.smithi165:> echo no | sudo yum reinstall kernel || true 2024-04-26T04:10:25.544 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:25.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:25.545 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalled: 2024-04-26T04:10:25.545 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-26T04:10:25.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:10:25.545 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:10:25.708 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T04:10:25.755 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-26T04:10:25.755 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-26T04:10:25.755 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-26T04:10:25.755 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-26T04:10:25.756 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T04:10:25.783 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-04-26T04:10:25.809 DEBUG:teuthology.parallel:result is None 2024-04-26T04:10:25.973 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:02 ago on Fri 26 Apr 2024 04:10:23 AM UTC. 2024-04-26T04:10:26.110 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalling: 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.117 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 255 k 2024-04-26T04:10:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 0 2024-04-26T04:10:26.118 INFO:teuthology.orchestra.run.smithi165.stderr:Operation aborted. 2024-04-26T04:10:26.177 INFO:teuthology.orchestra.run.smithi165.stdout:Is this ok [y/N]: 2024-04-26T04:10:26.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum reinstall -y kernel || true 2024-04-26T04:10:26.672 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Apr 2024 04:10:23 AM UTC. 2024-04-26T04:10:26.789 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalling: 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:10:26.790 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:26.791 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 255 k 2024-04-26T04:10:26.791 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 0 2024-04-26T04:10:26.791 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:10:27.367 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.0 MB/s | 255 kB 00:00 2024-04-26T04:10:27.368 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:10:27.368 INFO:teuthology.orchestra.run.smithi165.stdout:Total 442 kB/s | 255 kB 00:00 2024-04-26T04:10:27.383 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:10:27.567 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:10:27.568 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:10:27.573 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:10:27.573 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:10:27.710 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:10:27.785 INFO:teuthology.orchestra.run.smithi165.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T04:10:27.785 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:28.491 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:28.491 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T04:10:29.190 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T04:10:29.191 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:29.191 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalled: 2024-04-26T04:10:29.191 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-26T04:10:29.191 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:10:29.191 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:10:29.371 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T04:10:29.418 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-26T04:10:29.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-26T04:10:29.418 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-26T04:10:29.418 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-26T04:10:29.418 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T04:10:29.451 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-04-26T04:10:29.470 DEBUG:teuthology.parallel:result is None 2024-04-26T04:10:29.470 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T04:10:29.478 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T04:10:29.478 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T04:10:29.483 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T04:10:29.526 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T04:10:29.533 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T04:10:29.539 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T04:10:29.539 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T04:10:29.542 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T04:10:29.646 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T04:10:29.701 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T04:10:29.702 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T04:10:29.705 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T04:10:29.759 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:10:29.762 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:10:29.770 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:10:29.772 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:10:29.774 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T04:10:29.781 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T04:10:29.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T04:10:29.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T04:10:29.808 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T04:10:29.817 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T04:10:29.818 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T04:10:29.852 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T04:10:29.866 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:10:29.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:10:29.998 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:10:30.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:10:30.122 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:10:30.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T04:10:30.187 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:10:30.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:10:30.271 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:10:30.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:10:30.392 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:10:30.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T04:10:30.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-26T04:10:30.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-04-26T04:10:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T04:10:30.529 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T04:10:31.047 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T04:10:31.054 INFO:teuthology.task.internal:Starting timer... 2024-04-26T04:10:31.055 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T04:10:31.068 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T04:10:31.076 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-26T04:10:31.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-04-26T04:10:31.157 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-04-26T04:10:31.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-04-26T04:10:31.208 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-04-26T04:10:31.210 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T04:10:31.210 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-04-26T04:10:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-04-26T04:10:31.235 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-04-26T04:10:31.260 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-04-26T04:10:31.260 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-04-26T04:10:31.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-26T04:10:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:10:31.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-26T04:10:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:10:31.326 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T04:10:31.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-04-26T04:10:31.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-04-26T04:10:31.390 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T04:10:31.400 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T04:10:31.417 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T04:10:31.418 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T04:13:59.348 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-04-26T04:13:59.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-26T04:13:59.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:13:59.426 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-26T04:13:59.506 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-26T04:13:59.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-26T04:13:59.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:13:59.581 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-04-26T04:13:59.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-26T04:13:59.662 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T04:13:59.673 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T04:13:59.673 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T04:13:59.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:13:59.676 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T04:13:59.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:13:59.702 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T04:13:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T04:13:59.732 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T04:13:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-04-26T04:13:59.745 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T04:13:59.749 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-04-26T04:13:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T04:13:59.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-04-26T04:13:59.775 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T04:13:59.779 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-04-26T04:13:59.792 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T04:13:59.805 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T04:13:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-26T04:13:59.834 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T04:13:59.834 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-26T04:13:59.835 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.835 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.835 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.835 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.858 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-26T04:13:59.860 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T04:13:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-26T04:13:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.861 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:13:59.863 INFO:teuthology.run_tasks:Running task install... 2024-04-26T04:13:59.870 DEBUG:teuthology.task.install:project ceph 2024-04-26T04:13:59.871 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}} 2024-04-26T04:13:59.871 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'} 2024-04-26T04:13:59.871 INFO:teuthology.task.install:Using flavor: default 2024-04-26T04:13:59.876 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T04:13:59.877 INFO:teuthology.task.install:extra packages: [] 2024-04-26T04:13:59.877 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'tag': None, 'wait_for_package': False} 2024-04-26T04:13:59.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:13:59.879 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'tag': None, 'wait_for_package': False} 2024-04-26T04:13:59.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:14:00.034 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/ 2024-04-26T04:14:00.034 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2457.gddeaa917 2024-04-26T04:14:00.101 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/ 2024-04-26T04:14:00.101 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2457.gddeaa917 2024-04-26T04:14:00.151 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T04:14:00.151 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:14:00.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T04:14:00.177 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-26T04:14:00.177 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ddeaa91797339a19172c3036ff48cca58c12f448/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T04:14:00.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T04:14:00.274 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T04:14:00.274 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:14:00.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T04:14:00.301 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-26T04:14:00.301 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ddeaa91797339a19172c3036ff48cca58c12f448/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T04:14:00.316 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T04:14:00.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T04:14:00.379 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-04-26T04:14:00.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-04-26T04:14:00.440 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T04:14:00.503 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-04-26T04:14:00.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-04-26T04:14:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:48 files removed 2024-04-26T04:14:00.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-04-26T04:14:00.730 INFO:teuthology.orchestra.run.smithi165.stdout:48 files removed 2024-04-26T04:14:00.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw 2024-04-26T04:14:01.300 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-26T04:14:01.477 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-04-26T04:14:01.734 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-26T04:14:01.903 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-26T04:14:02.077 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-26T04:14:02.345 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-26T04:14:02.696 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-26T04:14:03.122 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-26T04:14:05.564 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-26T04:14:06.151 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-04-26T04:14:10.149 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.2 MB 00:01 2024-04-26T04:14:10.391 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-26T04:14:11.789 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-26T04:14:11.813 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-26T04:14:12.865 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-26T04:14:13.281 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T04:14:17.957 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-26T04:14:18.314 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-26T04:14:20.116 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:14:20.118 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:14:20.118 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:20.118 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:14:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 10 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.4 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.1 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 5.4 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 21 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 25 k 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T04:14:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 746 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 500 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 5.3 M 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 45 k 2024-04-26T04:14:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 133 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 161 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 319 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 298 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 100 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T04:14:20.121 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-04-26T04:14:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:20.123 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 60 M 2024-04-26T04:14:20.123 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:14:20.518 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 10 M 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.4 M 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.1 M 2024-04-26T04:14:20.520 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 5.4 M 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 21 M 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 25 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 746 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T04:14:20.521 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 500 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 5.3 M 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 45 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 133 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 161 k 2024-04-26T04:14:20.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 319 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 298 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 100 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:20.523 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout:Install 30 Packages 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 60 M 2024-04-26T04:14:20.524 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:14:21.625 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-common-19.0.0-2457.gddeaa917.el9.x 16 MB/s | 21 MB 00:01 2024-04-26T04:14:21.726 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-selinux-19.0.0-2457.gddeaa917.el9. 252 kB/s | 25 kB 00:00 2024-04-26T04:14:21.992 INFO:teuthology.orchestra.run.smithi165.stdout:(1/32): ceph-base-19.0.0-2457.gddeaa917.el9.x86 6.9 MB/s | 5.4 MB 00:00 2024-04-26T04:14:22.043 INFO:teuthology.orchestra.run.smithi165.stdout:(2/32): ceph-selinux-19.0.0-2457.gddeaa917.el9. 505 kB/s | 25 kB 00:00 2024-04-26T04:14:22.151 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): ceph-base-19.0.0-2457.gddeaa917.el9.x86 2.9 MB/s | 5.4 MB 00:01 2024-04-26T04:14:22.151 INFO:teuthology.orchestra.run.smithi165.stdout:(3/32): libcephfs2-19.0.0-2457.gddeaa917.el9.x8 6.7 MB/s | 746 kB 00:00 2024-04-26T04:14:22.243 INFO:teuthology.orchestra.run.smithi165.stdout:(4/32): libradosstriper1-19.0.0-2457.gddeaa917. 5.4 MB/s | 500 kB 00:00 2024-04-26T04:14:22.251 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libcephfs2-19.0.0-2457.gddeaa917.el9.x8 1.4 MB/s | 746 kB 00:00 2024-04-26T04:14:22.301 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): libradosstriper1-19.0.0-2457.gddeaa917. 3.3 MB/s | 500 kB 00:00 2024-04-26T04:14:22.601 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): python3-ceph-argparse-19.0.0-2457.gddea 150 kB/s | 45 kB 00:00 2024-04-26T04:14:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-common-19.0.0-2457.gddeaa9 1.2 MB/s | 133 kB 00:00 2024-04-26T04:14:22.818 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): python3-cephfs-19.0.0-2457.gddeaa917.el 1.5 MB/s | 161 kB 00:00 2024-04-26T04:14:22.902 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): librgw2-19.0.0-2457.gddeaa917.el9.x86_6 8.2 MB/s | 5.3 MB 00:00 2024-04-26T04:14:22.952 INFO:teuthology.orchestra.run.smithi165.stdout:(5/32): ceph-radosgw-19.0.0-2457.gddeaa917.el9. 5.9 MB/s | 10 MB 00:01 2024-04-26T04:14:22.952 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rbd-19.0.0-2457.gddeaa917.el9. 5.8 MB/s | 298 kB 00:00 2024-04-26T04:14:22.978 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): python3-rados-19.0.0-2457.gddeaa917.el 2.0 MB/s | 319 kB 00:00 2024-04-26T04:14:23.003 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): python3-rgw-19.0.0-2457.gddeaa917.el9. 1.9 MB/s | 100 kB 00:00 2024-04-26T04:14:23.027 INFO:teuthology.orchestra.run.smithi165.stdout:(6/32): librgw2-19.0.0-2457.gddeaa917.el9.x86_6 6.8 MB/s | 5.3 MB 00:00 2024-04-26T04:14:23.052 INFO:teuthology.orchestra.run.smithi165.stdout:(7/32): python3-ceph-argparse-19.0.0-2457.gddea 452 kB/s | 45 kB 00:00 2024-04-26T04:14:23.077 INFO:teuthology.orchestra.run.smithi165.stdout:(8/32): python3-ceph-common-19.0.0-2457.gddeaa9 2.6 MB/s | 133 kB 00:00 2024-04-26T04:14:23.102 INFO:teuthology.orchestra.run.smithi165.stdout:(9/32): python3-cephfs-19.0.0-2457.gddeaa917.el 3.1 MB/s | 161 kB 00:00 2024-04-26T04:14:23.136 INFO:teuthology.orchestra.run.smithi165.stdout:(10/32): python3-rados-19.0.0-2457.gddeaa917.el 5.3 MB/s | 319 kB 00:00 2024-04-26T04:14:23.161 INFO:teuthology.orchestra.run.smithi165.stdout:(11/32): python3-rbd-19.0.0-2457.gddeaa917.el9. 5.0 MB/s | 298 kB 00:00 2024-04-26T04:14:23.170 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 961 kB/s | 160 kB 00:00 2024-04-26T04:14:23.196 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 493 kB/s | 107 kB 00:00 2024-04-26T04:14:23.196 INFO:teuthology.orchestra.run.smithi165.stdout:(12/32): python3-rgw-19.0.0-2457.gddeaa917.el9. 1.7 MB/s | 100 kB 00:00 2024-04-26T04:14:23.220 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-26T04:14:23.329 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): ceph-radosgw-19.0.0-2457.gddeaa917.el9 3.4 MB/s | 10 MB 00:03 2024-04-26T04:14:23.362 INFO:teuthology.orchestra.run.smithi165.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 534 kB/s | 107 kB 00:00 2024-04-26T04:14:23.388 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-04-26T04:14:23.412 INFO:teuthology.orchestra.run.smithi165.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 738 kB/s | 160 kB 00:00 2024-04-26T04:14:23.413 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-04-26T04:14:23.429 INFO:teuthology.orchestra.run.smithi165.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-04-26T04:14:23.438 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 392 kB/s | 42 kB 00:00 2024-04-26T04:14:23.464 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-04-26T04:14:23.489 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 721 kB/s | 36 kB 00:00 2024-04-26T04:14:23.514 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 632 kB/s | 47 kB 00:00 2024-04-26T04:14:23.538 INFO:teuthology.orchestra.run.smithi165.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-26T04:14:23.563 INFO:teuthology.orchestra.run.smithi165.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-26T04:14:23.588 INFO:teuthology.orchestra.run.smithi165.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-04-26T04:14:23.643 INFO:teuthology.orchestra.run.smithi165.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.7 MB/s | 221 kB 00:00 2024-04-26T04:14:23.664 INFO:teuthology.orchestra.run.smithi165.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 481 kB/s | 36 kB 00:00 2024-04-26T04:14:23.746 INFO:teuthology.orchestra.run.smithi165.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 460 kB/s | 47 kB 00:00 2024-04-26T04:14:23.748 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-04-26T04:14:23.898 INFO:teuthology.orchestra.run.smithi165.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-04-26T04:14:23.923 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 673 kB/s | 309 kB 00:00 2024-04-26T04:14:23.939 INFO:teuthology.orchestra.run.smithi165.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-04-26T04:14:23.949 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 242 kB/s | 49 kB 00:00 2024-04-26T04:14:23.981 INFO:teuthology.orchestra.run.smithi165.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-26T04:14:24.032 INFO:teuthology.orchestra.run.smithi165.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-26T04:14:24.099 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 383 kB/s | 67 kB 00:00 2024-04-26T04:14:24.200 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-26T04:14:24.215 INFO:teuthology.orchestra.run.smithi165.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-04-26T04:14:24.291 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-04-26T04:14:24.307 INFO:teuthology.orchestra.run.smithi165.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-26T04:14:24.475 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.6 MB/s | 839 kB 00:00 2024-04-26T04:14:24.517 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-26T04:14:24.582 INFO:teuthology.orchestra.run.smithi165.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-26T04:14:24.675 INFO:teuthology.orchestra.run.smithi165.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-04-26T04:14:24.784 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): librbd1-19.0.0-2457.gddeaa917.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-04-26T04:14:24.983 INFO:teuthology.orchestra.run.smithi165.stdout:(30/32): librados2-19.0.0-2457.gddeaa917.el9.x8 8.5 MB/s | 3.4 MB 00:00 2024-04-26T04:14:25.100 INFO:teuthology.orchestra.run.smithi165.stdout:(31/32): librbd1-19.0.0-2457.gddeaa917.el9.x86_ 7.4 MB/s | 3.1 MB 00:00 2024-04-26T04:14:25.643 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): librados2-19.0.0-2457.gddeaa917.el9.x8 2.5 MB/s | 3.4 MB 00:01 2024-04-26T04:14:25.647 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:14:25.647 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 60 MB 00:05 2024-04-26T04:14:25.751 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:14:25.788 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:14:25.788 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:14:26.250 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:14:26.251 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:14:27.702 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:14:27.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-26T04:14:27.822 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-26T04:14:27.847 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-26T04:14:27.870 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-26T04:14:27.898 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-26T04:14:28.096 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-26T04:14:28.150 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-26T04:14:28.546 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-26T04:14:28.600 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:28.691 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:28.779 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 8/34 2024-04-26T04:14:28.809 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 9/34 2024-04-26T04:14:28.872 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 9/34 2024-04-26T04:14:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-26T04:14:28.944 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9. 11/34 2024-04-26T04:14:29.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 12/34 2024-04-26T04:14:29.036 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 13/34 2024-04-26T04:14:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 13/34 2024-04-26T04:14:29.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-26T04:14:29.250 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-26T04:14:29.275 INFO:teuthology.orchestra.run.smithi165.stdout:(32/32): ceph-common-19.0.0-2457.gddeaa917.el9. 2.6 MB/s | 21 MB 00:08 2024-04-26T04:14:29.280 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:14:29.280 INFO:teuthology.orchestra.run.smithi165.stdout:Total 6.8 MB/s | 60 MB 00:08 2024-04-26T04:14:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-26T04:14:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-26T04:14:29.351 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-26T04:14:29.380 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-26T04:14:29.384 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:14:29.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-26T04:14:29.421 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:14:29.421 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:14:29.726 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-26T04:14:29.767 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 22/34 2024-04-26T04:14:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 22/34 2024-04-26T04:14:29.886 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:14:29.886 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:14:29.888 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 23/34 2024-04-26T04:14:29.924 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-26T04:14:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-26T04:14:30.460 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 26/34 2024-04-26T04:14:30.515 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 26/34 2024-04-26T04:14:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 27/34 2024-04-26T04:14:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x8 28/34 2024-04-26T04:14:31.398 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:14:31.492 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-26T04:14:31.529 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-26T04:14:31.563 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-26T04:14:31.586 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-26T04:14:31.608 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-26T04:14:31.802 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-26T04:14:31.866 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-26T04:14:32.247 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-26T04:14:32.298 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:32.388 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:32.477 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 8/34 2024-04-26T04:14:32.508 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 9/34 2024-04-26T04:14:32.570 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 9/34 2024-04-26T04:14:32.599 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-26T04:14:32.642 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9. 11/34 2024-04-26T04:14:32.711 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 12/34 2024-04-26T04:14:32.735 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 13/34 2024-04-26T04:14:32.799 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 13/34 2024-04-26T04:14:32.833 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-26T04:14:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:32.919 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-26T04:14:32.975 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-26T04:14:33.007 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-26T04:14:33.032 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-26T04:14:33.061 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-26T04:14:33.094 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-26T04:14:33.408 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-26T04:14:33.457 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 22/34 2024-04-26T04:14:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:33.547 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 22/34 2024-04-26T04:14:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 30/34 2024-04-26T04:14:33.578 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 23/34 2024-04-26T04:14:33.614 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-26T04:14:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 30/34 2024-04-26T04:14:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T04:14:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T04:14:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:33.657 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:34.086 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-26T04:14:34.117 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 26/34 2024-04-26T04:14:34.188 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 26/34 2024-04-26T04:14:34.257 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 27/34 2024-04-26T04:14:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x8 28/34 2024-04-26T04:14:36.614 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:36.658 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:37.198 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 29/34 2024-04-26T04:14:37.235 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 30/34 2024-04-26T04:14:37.328 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 30/34 2024-04-26T04:14:37.328 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T04:14:37.328 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T04:14:37.329 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:37.362 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:43.213 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-04-26T04:14:43.214 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:44.176 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 32/34 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 32/34 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:14:44.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:44.205 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-26T04:14:44.295 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-26T04:14:44.297 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 1/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 3/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 4/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 5/34 2024-04-26T04:14:45.474 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 6/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9. 8/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x8 9/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 10/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 11/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 12/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 13/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-26T04:14:45.475 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-26T04:14:45.476 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-26T04:14:45.477 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-26T04:14:45.477 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-26T04:14:45.477 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-26T04:14:45.478 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-26T04:14:45.478 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:45.478 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-26T04:14:45.478 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 33/34 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.268 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.269 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:46.270 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:46.271 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:14:46.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-04-26T04:14:46.814 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:47.088 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:30 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:14:47.211 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 48 M 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 164 k 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:47.212 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-04-26T04:14:47.213 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:47.213 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 49 M 2024-04-26T04:14:47.213 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 207 M 2024-04-26T04:14:47.213 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:14:47.504 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 305 kB 00:00 2024-04-26T04:14:47.546 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-26T04:14:47.613 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): libcephsqlite-19.0.0-2457.gddeaa917.el9. 576 kB/s | 164 kB 00:00 2024-04-26T04:14:47.777 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 32/34 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 32/34 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:14:47.801 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:47.803 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-26T04:14:47.904 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-26T04:14:47.904 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:49.037 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:49.037 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 1/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 3/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 4/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 5/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_6 6/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 7/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9. 8/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x8 9/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 10/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 11/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 12/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 13/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-26T04:14:49.038 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-26T04:14:49.039 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-26T04:14:49.040 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-26T04:14:49.040 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-26T04:14:49.040 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-26T04:14:49.040 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 31/34 2024-04-26T04:14:49.040 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-26T04:14:49.041 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 33/34 2024-04-26T04:14:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-26T04:14:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:49.841 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:49.844 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:14:50.008 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-19.0.0-2457.gddeaa917.el9.x86_ 18 MB/s | 48 MB 00:02 2024-04-26T04:14:50.011 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:14:50.011 INFO:teuthology.orchestra.run.smithi098.stdout:Total 17 MB/s | 49 MB 00:02 2024-04-26T04:14:50.038 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:14:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:14:50.059 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:14:50.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-test 2024-04-26T04:14:50.425 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:14:50.425 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:14:50.642 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:32 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:14:50.766 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 48 M 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:14:50.767 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 164 k 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-04-26T04:14:50.768 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:14:50.769 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 49 M 2024-04-26T04:14:50.769 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 207 M 2024-04-26T04:14:50.769 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:14:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:14:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-26T04:14:51.082 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-26T04:14:51.086 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 305 kB 00:00 2024-04-26T04:14:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 3/4 2024-04-26T04:14:51.145 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T04:14:51.312 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): libcephsqlite-19.0.0-2457.gddeaa917.el9. 411 kB/s | 164 kB 00:00 2024-04-26T04:14:53.556 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): ceph-test-19.0.0-2457.gddeaa917.el9.x86_ 18 MB/s | 48 MB 00:02 2024-04-26T04:14:53.559 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:14:53.559 INFO:teuthology.orchestra.run.smithi165.stdout:Total 17 MB/s | 49 MB 00:02 2024-04-26T04:14:53.586 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:14:53.607 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:14:53.607 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:14:53.974 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:14:53.975 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:14:54.652 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:14:54.723 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-26T04:14:54.771 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-26T04:14:54.799 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 3/4 2024-04-26T04:14:55.648 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 3/4 2024-04-26T04:14:55.723 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 4/4 2024-04-26T04:14:58.020 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 4/4 2024-04-26T04:14:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 1/4 2024-04-26T04:14:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 2/4 2024-04-26T04:14:58.021 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:58.284 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:14:58.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-04-26T04:14:59.081 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:42 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:14:59.206 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 6.6 k 2024-04-26T04:14:59.209 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 2.4 M 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 1.5 M 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 249 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 4.7 M 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 17 M 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T04:14:59.210 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T04:14:59.211 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T04:14:59.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:Install 36 Packages 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-04-26T04:14:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 107 M 2024-04-26T04:14:59.214 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:14:59.333 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 3/4 2024-04-26T04:14:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 4/4 2024-04-26T04:14:59.810 INFO:teuthology.orchestra.run.smithi098.stdout:(1/36): ceph-19.0.0-2457.gddeaa917.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-04-26T04:14:59.977 INFO:teuthology.orchestra.run.smithi098.stdout:(2/36): ceph-mgr-19.0.0-2457.gddeaa917.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-04-26T04:15:00.277 INFO:teuthology.orchestra.run.smithi098.stdout:(3/36): ceph-mds-19.0.0-2457.gddeaa917.el9.x86_ 3.5 MB/s | 2.4 MB 00:00 2024-04-26T04:15:00.377 INFO:teuthology.orchestra.run.smithi098.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2457.gddea 2.4 MB/s | 249 kB 00:00 2024-04-26T04:15:00.503 INFO:teuthology.orchestra.run.smithi098.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-04-26T04:15:00.579 INFO:teuthology.orchestra.run.smithi098.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 16 MB/s | 1.2 MB 00:00 2024-04-26T04:15:00.620 INFO:teuthology.orchestra.run.smithi098.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-26T04:15:00.687 INFO:teuthology.orchestra.run.smithi098.stdout:(8/36): ceph-mon-19.0.0-2457.gddeaa917.el9.x86_ 5.4 MB/s | 4.7 MB 00:00 2024-04-26T04:15:00.712 INFO:teuthology.orchestra.run.smithi098.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-04-26T04:15:00.737 INFO:teuthology.orchestra.run.smithi098.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.5 MB/s | 125 kB 00:00 2024-04-26T04:15:00.763 INFO:teuthology.orchestra.run.smithi098.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-04-26T04:15:00.788 INFO:teuthology.orchestra.run.smithi098.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-26T04:15:00.813 INFO:teuthology.orchestra.run.smithi098.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-26T04:15:00.830 INFO:teuthology.orchestra.run.smithi098.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 835 kB/s | 35 kB 00:00 2024-04-26T04:15:00.847 INFO:teuthology.orchestra.run.smithi098.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-04-26T04:15:00.864 INFO:teuthology.orchestra.run.smithi098.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-26T04:15:00.881 INFO:teuthology.orchestra.run.smithi098.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 659 kB/s | 22 kB 00:00 2024-04-26T04:15:01.165 INFO:teuthology.orchestra.run.smithi098.stdout:(18/36): ceph-osd-19.0.0-2457.gddeaa917.el9.x86 14 MB/s | 17 MB 00:01 2024-04-26T04:15:01.182 INFO:teuthology.orchestra.run.smithi098.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-04-26T04:15:01.266 INFO:teuthology.orchestra.run.smithi098.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2024-04-26T04:15:01.308 INFO:teuthology.orchestra.run.smithi098.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-26T04:15:01.324 INFO:teuthology.orchestra.run.smithi098.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-04-26T04:15:01.358 INFO:teuthology.orchestra.run.smithi098.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-04-26T04:15:01.391 INFO:teuthology.orchestra.run.smithi098.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-04-26T04:15:01.408 INFO:teuthology.orchestra.run.smithi098.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-04-26T04:15:01.450 INFO:teuthology.orchestra.run.smithi098.stdout:(26/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-04-26T04:15:01.467 INFO:teuthology.orchestra.run.smithi098.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-04-26T04:15:01.484 INFO:teuthology.orchestra.run.smithi098.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-04-26T04:15:01.509 INFO:teuthology.orchestra.run.smithi098.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-04-26T04:15:01.535 INFO:teuthology.orchestra.run.smithi098.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-04-26T04:15:01.577 INFO:teuthology.orchestra.run.smithi098.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-04-26T04:15:01.602 INFO:teuthology.orchestra.run.smithi098.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-04-26T04:15:01.626 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 4/4 2024-04-26T04:15:01.626 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 1/4 2024-04-26T04:15:01.626 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 2/4 2024-04-26T04:15:01.626 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-26T04:15:01.652 INFO:teuthology.orchestra.run.smithi098.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T04:15:01.711 INFO:teuthology.orchestra.run.smithi098.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-04-26T04:15:01.736 INFO:teuthology.orchestra.run.smithi098.stdout:(35/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-04-26T04:15:01.770 INFO:teuthology.orchestra.run.smithi098.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-04-26T04:15:01.774 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:01.774 INFO:teuthology.orchestra.run.smithi098.stdout:Total 12 MB/s | 30 MB 00:02 2024-04-26T04:15:01.911 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:01.923 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:01.950 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:01.951 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:02.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph 2024-04-26T04:15:02.244 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:02.245 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:02.696 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:44 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:02.820 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 6.6 k 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 2.4 M 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 1.5 M 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 249 k 2024-04-26T04:15:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 4.7 M 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 17 M 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T04:15:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T04:15:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout:Install 36 Packages 2024-04-26T04:15:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:02.827 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 30 M 2024-04-26T04:15:02.827 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 107 M 2024-04-26T04:15:02.827 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:03.490 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-26T04:15:03.605 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-26T04:15:03.633 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-26T04:15:03.704 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-26T04:15:03.749 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-26T04:15:03.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-26T04:15:03.875 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-26T04:15:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-26T04:15:03.988 INFO:teuthology.orchestra.run.smithi165.stdout:(1/36): ceph-19.0.0-2457.gddeaa917.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-04-26T04:15:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-26T04:15:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-26T04:15:04.066 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-26T04:15:04.101 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-26T04:15:04.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-26T04:15:04.183 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-26T04:15:04.188 INFO:teuthology.orchestra.run.smithi165.stdout:(2/36): ceph-mgr-19.0.0-2457.gddeaa917.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-04-26T04:15:04.218 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-26T04:15:04.230 INFO:teuthology.orchestra.run.smithi165.stdout:(3/36): ceph-mds-19.0.0-2457.gddeaa917.el9.x86_ 5.4 MB/s | 2.4 MB 00:00 2024-04-26T04:15:04.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-26T04:15:04.274 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-26T04:15:04.348 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-26T04:15:04.380 INFO:teuthology.orchestra.run.smithi165.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2457.gddea 1.6 MB/s | 249 kB 00:00 2024-04-26T04:15:04.488 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-26T04:15:04.497 INFO:teuthology.orchestra.run.smithi165.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-26T04:15:04.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-26T04:15:04.572 INFO:teuthology.orchestra.run.smithi165.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 16 MB/s | 1.2 MB 00:00 2024-04-26T04:15:04.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-26T04:15:04.631 INFO:teuthology.orchestra.run.smithi165.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-26T04:15:04.658 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-26T04:15:04.698 INFO:teuthology.orchestra.run.smithi165.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-26T04:15:04.715 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-26T04:15:04.748 INFO:teuthology.orchestra.run.smithi165.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-04-26T04:15:04.771 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-26T04:15:04.824 INFO:teuthology.orchestra.run.smithi165.stdout:(10/36): ceph-mon-19.0.0-2457.gddeaa917.el9.x86 5.6 MB/s | 4.7 MB 00:00 2024-04-26T04:15:04.848 INFO:teuthology.orchestra.run.smithi165.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-04-26T04:15:04.874 INFO:teuthology.orchestra.run.smithi165.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-26T04:15:04.899 INFO:teuthology.orchestra.run.smithi165.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-26T04:15:04.924 INFO:teuthology.orchestra.run.smithi165.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 694 kB/s | 35 kB 00:00 2024-04-26T04:15:04.949 INFO:teuthology.orchestra.run.smithi165.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T04:15:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-26T04:15:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-26T04:15:05.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-26T04:15:05.133 INFO:teuthology.orchestra.run.smithi165.stdout:(16/36): ceph-osd-19.0.0-2457.gddeaa917.el9.x86 18 MB/s | 17 MB 00:00 2024-04-26T04:15:05.158 INFO:teuthology.orchestra.run.smithi165.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 178 kB/s | 42 kB 00:00 2024-04-26T04:15:05.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-26T04:15:05.184 INFO:teuthology.orchestra.run.smithi165.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-04-26T04:15:05.228 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-26T04:15:05.342 INFO:teuthology.orchestra.run.smithi165.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-04-26T04:15:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-26T04:15:05.403 INFO:teuthology.orchestra.run.smithi165.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-04-26T04:15:05.426 INFO:teuthology.orchestra.run.smithi165.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2024-04-26T04:15:05.459 INFO:teuthology.orchestra.run.smithi165.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-04-26T04:15:05.485 INFO:teuthology.orchestra.run.smithi165.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-04-26T04:15:05.501 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9. 31/36 2024-04-26T04:15:05.518 INFO:teuthology.orchestra.run.smithi165.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-04-26T04:15:05.530 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 32/36 2024-04-26T04:15:05.544 INFO:teuthology.orchestra.run.smithi165.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 32/36 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T04:15:05.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:05.569 INFO:teuthology.orchestra.run.smithi165.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-04-26T04:15:05.594 INFO:teuthology.orchestra.run.smithi165.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-04-26T04:15:05.619 INFO:teuthology.orchestra.run.smithi165.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-04-26T04:15:05.645 INFO:teuthology.orchestra.run.smithi165.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-26T04:15:05.678 INFO:teuthology.orchestra.run.smithi165.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-04-26T04:15:05.703 INFO:teuthology.orchestra.run.smithi165.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-26T04:15:05.737 INFO:teuthology.orchestra.run.smithi165.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 975 kB/s | 90 kB 00:00 2024-04-26T04:15:05.762 INFO:teuthology.orchestra.run.smithi165.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-04-26T04:15:05.788 INFO:teuthology.orchestra.run.smithi165.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T04:15:05.821 INFO:teuthology.orchestra.run.smithi165.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-26T04:15:05.880 INFO:teuthology.orchestra.run.smithi165.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-04-26T04:15:05.883 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:05.884 INFO:teuthology.orchestra.run.smithi165.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-04-26T04:15:06.023 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:06.063 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:06.063 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:06.355 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:06.356 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:06.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 33/36 2024-04-26T04:15:07.007 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 33/36 2024-04-26T04:15:07.007 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:07.008 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T04:15:07.008 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T04:15:07.008 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T04:15:07.008 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:07.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 34/36 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 34/36 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T04:15:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:07.600 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:07.657 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-26T04:15:07.690 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-26T04:15:07.699 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 35/36 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 35/36 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T04:15:07.722 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:07.728 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-26T04:15:07.787 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 36/36 2024-04-26T04:15:07.798 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-26T04:15:07.884 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-26T04:15:07.916 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-26T04:15:08.008 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-26T04:15:08.086 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-26T04:15:08.127 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-26T04:15:08.162 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-26T04:15:08.183 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-26T04:15:08.209 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-26T04:15:08.251 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-26T04:15:08.292 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-26T04:15:08.327 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-26T04:15:08.361 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-26T04:15:08.399 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-26T04:15:08.474 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-26T04:15:08.612 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-26T04:15:08.691 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-26T04:15:08.751 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-26T04:15:08.789 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-26T04:15:08.839 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-26T04:15:08.894 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-26T04:15:09.083 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-26T04:15:09.135 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-26T04:15:09.214 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-26T04:15:09.311 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-26T04:15:09.351 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-26T04:15:09.468 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 36/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 1/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 2/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 3/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 4/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 5/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9. 6/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-26T04:15:09.547 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-26T04:15:09.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-26T04:15:09.549 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-26T04:15:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9. 31/36 2024-04-26T04:15:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 32/36 2024-04-26T04:15:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 32/36 2024-04-26T04:15:09.685 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:09.685 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T04:15:09.685 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T04:15:09.686 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T04:15:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T04:15:10.426 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T04:15:10.427 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T04:15:10.428 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T04:15:10.428 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T04:15:10.428 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T04:15:10.428 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:10.428 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:10.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-04-26T04:15:11.179 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 33/36 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 33/36 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T04:15:11.208 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:11.305 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:54 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:11.384 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:11.424 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:11.425 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:11.426 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:11.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-04-26T04:15:11.646 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 34/36 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 34/36 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T04:15:11.670 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:11.897 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 35/36 2024-04-26T04:15:11.920 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 35/36 2024-04-26T04:15:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T04:15:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T04:15:11.921 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T04:15:11.921 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:11.984 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 36/36 2024-04-26T04:15:12.001 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:12.119 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 762 k 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:12.120 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 762 k 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 768 k 2024-04-26T04:15:12.121 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:12.974 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-2457.gddeaa917.el9.noarch.rpm 893 kB/s | 762 kB 00:00 2024-04-26T04:15:12.975 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:12.975 INFO:teuthology.orchestra.run.smithi098.stdout:Total 891 kB/s | 762 kB 00:00 2024-04-26T04:15:12.975 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:12.981 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:12.981 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:12.987 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:12.987 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:13.521 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:13.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:13.739 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 36/36 2024-04-26T04:15:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 1/36 2024-04-26T04:15:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 2/36 2024-04-26T04:15:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 3/36 2024-04-26T04:15:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 4/36 2024-04-26T04:15:13.740 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 5/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9. 6/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-26T04:15:13.741 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-26T04:15:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-26T04:15:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-26T04:15:13.742 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-26T04:15:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-26T04:15:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-26T04:15:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-26T04:15:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-26T04:15:13.743 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-26T04:15:13.744 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-26T04:15:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-26T04:15:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-26T04:15:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-26T04:15:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-26T04:15:13.745 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-26T04:15:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-26T04:15:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-26T04:15:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-26T04:15:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-26T04:15:13.746 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-26T04:15:13.987 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:14.258 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:14.259 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:14.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-04-26T04:15:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-26T04:15:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:14.614 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T04:15:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T04:15:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:14.617 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:14.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-base 2024-04-26T04:15:15.013 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:58 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:15.133 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 145 k 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:15.134 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:15:15.135 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:15.135 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-04-26T04:15:15.135 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 435 k 2024-04-26T04:15:15.135 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:15.490 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:57 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:15.558 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-2457.gddeaa9 343 kB/s | 145 kB 00:00 2024-04-26T04:15:15.558 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:15.558 INFO:teuthology.orchestra.run.smithi098.stdout:Total 341 kB/s | 145 kB 00:00 2024-04-26T04:15:15.559 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:15.566 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:15.566 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:15.567 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-base-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:15.609 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:15.609 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:15:15.610 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:15.622 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:15.622 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:15.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install cephadm 2024-04-26T04:15:15.770 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:15.859 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:15.883 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:15.883 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:15.883 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T04:15:15.883 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:16.173 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:58 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:16.295 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 762 k 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:16.297 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:16.298 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:15:16.298 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:16.298 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 762 k 2024-04-26T04:15:16.298 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 768 k 2024-04-26T04:15:16.298 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:16.763 INFO:teuthology.orchestra.run.smithi165.stdout:cephadm-19.0.0-2457.gddeaa917.el9.noarch.rpm 1.6 MB/s | 762 kB 00:00 2024-04-26T04:15:16.763 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.6 MB/s | 762 kB 00:00 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:16.764 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:16.765 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:16.769 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:16.769 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:16.774 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:16.775 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:16.849 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:16.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-04-26T04:15:17.293 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:17.374 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:17.487 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:00 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:17.568 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:17.611 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:17.612 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:17.612 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:17.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-04-26T04:15:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/1 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:17.963 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:18.188 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:01 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:18.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-immutable-object-cache 2024-04-26T04:15:18.307 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:18.308 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:18.308 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:18.308 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:18.308 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 3.5 M 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 32 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 15 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:18.309 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 8.6 M 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 103 M 2024-04-26T04:15:18.310 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:18.708 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:00 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:18.831 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 145 k 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:15:18.832 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:18.833 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 145 k 2024-04-26T04:15:18.833 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 435 k 2024-04-26T04:15:18.833 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:19.059 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-immutable-object-cache-19.0.0-2457.gddeaa9 642 kB/s | 145 kB 00:00 2024-04-26T04:15:19.061 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:19.061 INFO:teuthology.orchestra.run.smithi165.stdout:Total 636 kB/s | 145 kB 00:00 2024-04-26T04:15:19.061 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:19.061 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2457.gdd 169 kB/s | 32 kB 00:00 2024-04-26T04:15:19.068 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:19.069 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:19.128 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:19.128 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:19.270 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:19.294 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2457.gdde 36 kB/s | 15 kB 00:00 2024-04-26T04:15:19.335 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:19.360 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:19.360 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T04:15:19.360 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:19.453 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): ceph-mgr-dashboard-19.0.0-2457.gddeaa91 6.1 MB/s | 3.5 MB 00:00 2024-04-26T04:15:19.562 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.0 MB/s | 1.0 MB 00:00 2024-04-26T04:15:19.587 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 921 kB/s | 269 kB 00:00 2024-04-26T04:15:19.621 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.1 MB/s | 863 kB 00:00 2024-04-26T04:15:19.896 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-04-26T04:15:20.088 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-26T04:15:20.113 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 666 kB/s | 144 kB 00:00 2024-04-26T04:15:20.164 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-04-26T04:15:20.247 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917. 1/1 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:20.263 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:20.289 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-04-26T04:15:20.295 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:20.295 INFO:teuthology.orchestra.run.smithi098.stdout:Total 4.3 MB/s | 8.6 MB 00:01 2024-04-26T04:15:20.343 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:20.360 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:20.360 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:20.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr 2024-04-26T04:15:20.516 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:20.517 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:20.992 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:02 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:21.072 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-mgr-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-26T04:15:21.115 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:21.117 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:15:21.117 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-26T04:15:21.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-dashboard 2024-04-26T04:15:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-26T04:15:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-26T04:15:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-26T04:15:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-26T04:15:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-26T04:15:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-26T04:15:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-26T04:15:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9 10/12 2024-04-26T04:15:21.686 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:03 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:21.808 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:21.809 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:21.809 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:21.809 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 3.5 M 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 32 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 15 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T04:15:21.810 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 8.6 M 2024-04-26T04:15:21.811 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 103 M 2024-04-26T04:15:21.812 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:22.488 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2457.gdde 83 kB/s | 15 kB 00:00 2024-04-26T04:15:22.546 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2457.gdd 130 kB/s | 32 kB 00:00 2024-04-26T04:15:22.647 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-04-26T04:15:22.671 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el 11/12 2024-04-26T04:15:22.672 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-04-26T04:15:22.701 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 12/12 2024-04-26T04:15:22.705 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 15 MB/s | 863 kB 00:00 2024-04-26T04:15:22.864 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): ceph-mgr-dashboard-19.0.0-2457.gddeaa91 6.3 MB/s | 3.5 MB 00:00 2024-04-26T04:15:22.931 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-04-26T04:15:22.989 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-04-26T04:15:23.048 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-04-26T04:15:23.073 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 374 kB/s | 31 kB 00:00 2024-04-26T04:15:23.123 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.9 MB/s | 2.0 MB 00:00 2024-04-26T04:15:23.148 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-04-26T04:15:23.152 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:23.153 INFO:teuthology.orchestra.run.smithi165.stdout:Total 6.4 MB/s | 8.6 MB 00:01 2024-04-26T04:15:23.202 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:23.218 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:23.218 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:23.374 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:23.375 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:23.906 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:23.982 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-26T04:15:24.017 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-26T04:15:24.064 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 12/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el 1/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 2/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9 3/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-26T04:15:24.099 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-26T04:15:24.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-26T04:15:24.100 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-26T04:15:24.111 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-26T04:15:24.196 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-26T04:15:24.338 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-26T04:15:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-26T04:15:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T04:15:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:24.416 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-26T04:15:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-26T04:15:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-26T04:15:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9 10/12 2024-04-26T04:15:24.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-26T04:15:25.187 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:08 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:25.308 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:25.309 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-26T04:15:25.309 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:25.309 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-26T04:15:25.309 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:25.309 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 7.4 M 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T04:15:25.310 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-26T04:15:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-04-26T04:15:25.311 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:25.311 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-04-26T04:15:25.312 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-04-26T04:15:25.312 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:25.582 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el 11/12 2024-04-26T04:15:25.618 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 12/12 2024-04-26T04:15:25.956 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-26T04:15:26.006 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 803 kB 00:00 2024-04-26T04:15:26.031 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 441 kB/s | 33 kB 00:00 2024-04-26T04:15:26.115 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-04-26T04:15:26.141 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 169 kB/s | 18 kB 00:00 2024-04-26T04:15:26.165 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-26T04:15:26.274 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2024-04-26T04:15:26.300 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-04-26T04:15:26.450 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-26T04:15:26.475 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.4 MB/s | 250 kB 00:00 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 12/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el 1/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noa 2/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9 3/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-26T04:15:26.752 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-26T04:15:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-26T04:15:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-26T04:15:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-26T04:15:26.753 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-26T04:15:26.875 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T04:15:27.045 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:27.046 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:27.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-26T04:15:27.603 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.1 MB/s | 7.4 MB 00:01 2024-04-26T04:15:27.606 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:27.606 INFO:teuthology.orchestra.run.smithi098.stdout:Total 17 MB/s | 38 MB 00:02 2024-04-26T04:15:27.804 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:09 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:27.927 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:27.927 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 7.4 M 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T04:15:27.928 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-04-26T04:15:27.929 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:27.930 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 38 M 2024-04-26T04:15:27.930 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 209 M 2024-04-26T04:15:27.930 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:27.964 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:27.990 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:27.990 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:28.347 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-26T04:15:28.372 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-04-26T04:15:28.396 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:28.396 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:28.397 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 659 kB/s | 33 kB 00:00 2024-04-26T04:15:28.473 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 30 MB/s | 3.0 MB 00:00 2024-04-26T04:15:28.498 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 185 kB/s | 18 kB 00:00 2024-04-26T04:15:28.523 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-26T04:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 41 MB/s | 4.7 MB 00:00 2024-04-26T04:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.1 MB/s | 244 kB 00:00 2024-04-26T04:15:28.742 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-04-26T04:15:28.766 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-04-26T04:15:29.091 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-26T04:15:29.218 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-26T04:15:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-26T04:15:29.415 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-26T04:15:29.459 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-26T04:15:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-26T04:15:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-26T04:15:29.741 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 4.7 MB/s | 7.4 MB 00:01 2024-04-26T04:15:29.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-26T04:15:29.744 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:29.745 INFO:teuthology.orchestra.run.smithi165.stdout:Total 21 MB/s | 38 MB 00:01 2024-04-26T04:15:30.097 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:30.124 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:30.124 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:30.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-26T04:15:30.496 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-26T04:15:30.536 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:30.536 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:31.382 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:31.446 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-26T04:15:31.490 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-26T04:15:31.591 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-26T04:15:31.632 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-26T04:15:31.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-26T04:15:31.811 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-26T04:15:31.839 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-26T04:15:31.916 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-26T04:15:32.552 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-26T04:15:32.668 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-26T04:15:32.746 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-26T04:15:32.777 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 12/12 2024-04-26T04:15:33.884 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-26T04:15:34.923 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-26T04:15:34.954 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 12/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 12/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 1/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-26T04:15:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T04:15:35.840 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:35.841 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:36.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-04-26T04:15:36.740 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:19 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:36.861 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 49 k 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T04:15:36.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:36.864 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-04-26T04:15:36.865 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:36.865 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-04-26T04:15:36.865 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-04-26T04:15:36.865 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:37.435 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 12/12 2024-04-26T04:15:37.435 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa 1/12 2024-04-26T04:15:37.435 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-26T04:15:37.435 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-26T04:15:37.436 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-26T04:15:37.670 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 26 kB 00:00 2024-04-26T04:15:37.686 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 101 kB/s | 19 kB 00:00 2024-04-26T04:15:37.779 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-26T04:15:37.779 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:37.779 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:37.779 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:37.779 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:37.780 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:37.820 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-04-26T04:15:37.846 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 999 kB/s | 159 kB 00:00 2024-04-26T04:15:37.896 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-04-26T04:15:37.921 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 712 kB/s | 54 kB 00:00 2024-04-26T04:15:37.946 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): ceph-mgr-rook-19.0.0-2457.gddeaa917.el9 110 kB/s | 49 kB 00:00 2024-04-26T04:15:38.163 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-04-26T04:15:38.189 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-04-26T04:15:38.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-rook 2024-04-26T04:15:38.355 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 355 kB/s | 59 kB 00:00 2024-04-26T04:15:38.397 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 456 kB/s | 206 kB 00:00 2024-04-26T04:15:38.606 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-04-26T04:15:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 325 kB/s | 90 kB 00:00 2024-04-26T04:15:38.633 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:38.634 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-04-26T04:15:38.698 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:38.713 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:38.713 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:38.730 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:20 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:38.797 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:38.797 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:38.850 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:38.851 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:38.851 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 49 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T04:15:38.852 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout:Install 13 Packages 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:38.853 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.2 M 2024-04-26T04:15:38.854 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 26 M 2024-04-26T04:15:38.854 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:39.188 INFO:teuthology.orchestra.run.smithi165.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 26 kB 00:00 2024-04-26T04:15:39.213 INFO:teuthology.orchestra.run.smithi165.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 19 kB 00:00 2024-04-26T04:15:39.280 INFO:teuthology.orchestra.run.smithi165.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-26T04:15:39.304 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:39.306 INFO:teuthology.orchestra.run.smithi165.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T04:15:39.331 INFO:teuthology.orchestra.run.smithi165.stdout:(5/13): ceph-mgr-rook-19.0.0-2457.gddeaa917.el9 180 kB/s | 49 kB 00:00 2024-04-26T04:15:39.356 INFO:teuthology.orchestra.run.smithi165.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-04-26T04:15:39.381 INFO:teuthology.orchestra.run.smithi165.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 718 kB/s | 54 kB 00:00 2024-04-26T04:15:39.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-26T04:15:39.525 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-26T04:15:39.588 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-26T04:15:39.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-26T04:15:39.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-26T04:15:39.768 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-26T04:15:39.831 INFO:teuthology.orchestra.run.smithi165.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 32 kB/s | 15 kB 00:00 2024-04-26T04:15:39.882 INFO:teuthology.orchestra.run.smithi165.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 61 kB/s | 32 kB 00:00 2024-04-26T04:15:39.883 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-26T04:15:39.935 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-26T04:15:40.074 INFO:teuthology.orchestra.run.smithi165.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-26T04:15:40.182 INFO:teuthology.orchestra.run.smithi165.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 257 kB/s | 206 kB 00:00 2024-04-26T04:15:40.257 INFO:teuthology.orchestra.run.smithi165.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2024-04-26T04:15:40.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-26T04:15:40.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-26T04:15:40.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-26T04:15:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-26T04:15:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 13/13 2024-04-26T04:15:41.224 INFO:teuthology.orchestra.run.smithi165.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 751 kB/s | 1.0 MB 00:01 2024-04-26T04:15:41.229 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:41.230 INFO:teuthology.orchestra.run.smithi165.stdout:Total 948 kB/s | 2.2 MB 00:02 2024-04-26T04:15:41.297 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:41.313 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:41.314 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:41.398 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:41.399 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:41.544 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 13/13 2024-04-26T04:15:41.544 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 1/13 2024-04-26T04:15:41.544 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-26T04:15:41.544 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-26T04:15:41.544 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-26T04:15:41.545 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-26T04:15:41.847 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-26T04:15:41.847 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:41.847 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T04:15:41.848 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:41.849 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:41.900 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:42.073 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-26T04:15:42.087 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-04-26T04:15:42.134 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-26T04:15:42.188 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-26T04:15:42.226 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-26T04:15:42.268 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-26T04:15:42.362 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-26T04:15:42.478 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-26T04:15:42.527 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-26T04:15:42.591 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:25 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:42.712 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:42.712 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:42.712 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 141 k 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-04-26T04:15:42.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:42.714 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-04-26T04:15:42.714 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-04-26T04:15:42.714 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:43.093 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-26T04:15:43.194 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2457.gddeaa917.e 513 kB/s | 141 kB 00:00 2024-04-26T04:15:43.239 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-26T04:15:43.283 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-26T04:15:43.316 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-26T04:15:43.364 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-26T04:15:43.369 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-04-26T04:15:43.386 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 13/13 2024-04-26T04:15:43.394 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-04-26T04:15:43.444 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 689 kB/s | 51 kB 00:00 2024-04-26T04:15:43.936 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 6.0 MB 00:01 2024-04-26T04:15:43.938 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:43.938 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.7 MB/s | 7.0 MB 00:01 2024-04-26T04:15:44.008 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:44.018 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:44.018 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:44.114 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:44.115 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:44.208 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 13/13 2024-04-26T04:15:44.208 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 1/13 2024-04-26T04:15:44.208 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-26T04:15:44.208 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-26T04:15:44.208 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-26T04:15:44.209 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-26T04:15:44.416 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:44.518 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-26T04:15:44.518 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:44.518 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:44.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:44.518 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:44.519 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:44.584 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-26T04:15:44.637 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-26T04:15:44.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-cephadm 2024-04-26T04:15:45.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-26T04:15:45.224 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-26T04:15:45.257 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:27 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:45.286 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-26T04:15:45.317 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 6/6 2024-04-26T04:15:45.377 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 141 k 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T04:15:45.378 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:45.379 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 7.0 M 2024-04-26T04:15:45.380 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 33 M 2024-04-26T04:15:45.380 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 6/6 2024-04-26T04:15:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/6 2024-04-26T04:15:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-26T04:15:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-26T04:15:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-26T04:15:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-26T04:15:46.025 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-04-26T04:15:46.117 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2457.gddeaa917.e 459 kB/s | 141 kB 00:00 2024-04-26T04:15:46.243 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 467 kB/s | 58 kB 00:00 2024-04-26T04:15:46.268 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T04:15:46.289 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:46.290 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:46.351 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-26T04:15:46.376 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 385 kB/s | 51 kB 00:00 2024-04-26T04:15:46.378 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:46.379 INFO:teuthology.orchestra.run.smithi165.stdout:Total 7.0 MB/s | 7.0 MB 00:00 2024-04-26T04:15:46.450 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:46.461 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:46.462 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:46.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-04-26T04:15:46.557 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:46.558 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:46.889 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-26T04:15:47.069 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:30 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-26T04:15:47.190 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:47.191 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:47.191 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:47.191 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:47.191 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:47.191 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 890 k 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 970 k 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-04-26T04:15:47.192 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:47.610 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-26T04:15:47.622 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 487 kB/s | 80 kB 00:00 2024-04-26T04:15:47.697 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-26T04:15:47.769 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-26T04:15:47.799 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 6/6 2024-04-26T04:15:48.222 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-2457.gddeaa917.el9.x86_ 1.1 MB/s | 890 kB 00:00 2024-04-26T04:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:48.224 INFO:teuthology.orchestra.run.smithi098.stdout:Total 939 kB/s | 970 kB 00:01 2024-04-26T04:15:48.236 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:48.260 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:48.260 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:48.339 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:48.339 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:48.484 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 6/6 2024-04-26T04:15:48.485 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 1/6 2024-04-26T04:15:48.485 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-26T04:15:48.485 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-26T04:15:48.485 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-26T04:15:48.485 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-26T04:15:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-26T04:15:48.732 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-26T04:15:48.732 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:48.732 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:48.733 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2/2 2024-04-26T04:15:48.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-fuse 2024-04-26T04:15:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2/2 2024-04-26T04:15:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/2 2024-04-26T04:15:49.472 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:31 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:49.592 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:49.592 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:49.592 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 890 k 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-04-26T04:15:49.593 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:49.594 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 970 k 2024-04-26T04:15:49.594 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 2.8 M 2024-04-26T04:15:49.594 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-26T04:15:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:49.669 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-26T04:15:49.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:49.670 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:49.811 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 666 kB/s | 80 kB 00:00 2024-04-26T04:15:49.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-04-26T04:15:50.111 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): ceph-fuse-19.0.0-2457.gddeaa917.el9.x86_ 2.1 MB/s | 890 kB 00:00 2024-04-26T04:15:50.112 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:50.112 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.8 MB/s | 970 kB 00:00 2024-04-26T04:15:50.124 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:50.145 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:50.145 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:50.222 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:50.222 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:50.407 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:50.466 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:33 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:50.555 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 264 k 2024-04-26T04:15:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-04-26T04:15:50.591 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:50.592 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 341 k 2024-04-26T04:15:50.592 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.5 M 2024-04-26T04:15:50.592 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:50.626 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2/2 2024-04-26T04:15:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 601 kB/s | 77 kB 00:00 2024-04-26T04:15:51.262 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-volume-19.0.0-2457.gddeaa917.el9.no 653 kB/s | 264 kB 00:00 2024-04-26T04:15:51.263 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:51.264 INFO:teuthology.orchestra.run.smithi098.stdout:Total 506 kB/s | 341 kB 00:00 2024-04-26T04:15:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2/2 2024-04-26T04:15:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/2 2024-04-26T04:15:51.275 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:51.285 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:51.285 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:51.312 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:51.312 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:51.464 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:51.506 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-26T04:15:51.507 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:51.507 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:51.507 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-26T04:15:51.507 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:51.507 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:51.655 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-26T04:15:51.682 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2/2 2024-04-26T04:15:51.697 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2/2 2024-04-26T04:15:51.697 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:51.697 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T04:15:51.697 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:51.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-volume 2024-04-26T04:15:52.307 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:34 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:52.403 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 1/2 2024-04-26T04:15:52.432 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.0.0-2457.gddeaa917.el9 ceph-noarch 264 k 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-04-26T04:15:52.433 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:52.434 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 341 k 2024-04-26T04:15:52.434 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 1.5 M 2024-04-26T04:15:52.434 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:52.672 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-26T04:15:52.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:52.672 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:52.673 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:52.673 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T04:15:52.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:52.673 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:52.690 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 581 kB/s | 77 kB 00:00 2024-04-26T04:15:52.866 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): ceph-volume-19.0.0-2457.gddeaa917.el9.no 856 kB/s | 264 kB 00:00 2024-04-26T04:15:52.867 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:52.867 INFO:teuthology.orchestra.run.smithi165.stdout:Total 786 kB/s | 341 kB 00:00 2024-04-26T04:15:52.878 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:52.888 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:52.888 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:52.914 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:52.914 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:53.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-04-26T04:15:53.060 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:53.225 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-26T04:15:53.268 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2/2 2024-04-26T04:15:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2/2 2024-04-26T04:15:53.284 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:15:53.284 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T04:15:53.284 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:53.532 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:36 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:53.654 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:53.655 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-26T04:15:53.656 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:53.656 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 127 k 2024-04-26T04:15:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-04-26T04:15:53.657 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-04-26T04:15:53.658 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:53.878 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-2457.gddeaa917.el9.x86_64 570 kB/s | 127 kB 00:00 2024-04-26T04:15:53.879 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:53.879 INFO:teuthology.orchestra.run.smithi098.stdout:Total 566 kB/s | 127 kB 00:00 2024-04-26T04:15:53.879 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:53.884 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:53.885 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:53.947 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:53.948 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:53.984 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 1/2 2024-04-26T04:15:54.098 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:54.199 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.0.0-2457.gddeaa917.el9.noarch 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:54.283 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:54.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados-devel 2024-04-26T04:15:54.594 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:54.826 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:54.827 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:54.827 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:54.827 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:54.827 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:54.827 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:55.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-04-26T04:15:55.122 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:37 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:55.245 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:55.246 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:55.246 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-04-26T04:15:55.246 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 127 k 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 127 k 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 456 k 2024-04-26T04:15:55.247 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:55.593 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:38 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:55.595 INFO:teuthology.orchestra.run.smithi165.stdout:librados-devel-19.0.0-2457.gddeaa917.el9.x86_64 364 kB/s | 127 kB 00:00 2024-04-26T04:15:55.596 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:55.596 INFO:teuthology.orchestra.run.smithi165.stdout:Total 362 kB/s | 127 kB 00:00 2024-04-26T04:15:55.596 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:55.602 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:55.602 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:55.665 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:55.665 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:55.672 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:55.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-04-26T04:15:55.815 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:55.908 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:56.302 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:56.322 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:39 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:56.443 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:56.443 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 32 k 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:15:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 32 k 2024-04-26T04:15:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 142 k 2024-04-26T04:15:56.445 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:56.581 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:56.616 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-2457.gddeaa917.el9.x86_6 185 kB/s | 32 kB 00:00 2024-04-26T04:15:56.616 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:56.617 INFO:teuthology.orchestra.run.smithi098.stdout:Total 184 kB/s | 32 kB 00:00 2024-04-26T04:15:56.617 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:15:56.621 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:15:56.621 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:15:56.623 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:15:56.623 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:15:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:15:56.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs2 2024-04-26T04:15:56.820 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:57.149 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:57.320 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:39 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:15:57.350 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:57.401 INFO:teuthology.orchestra.run.smithi165.stdout:Package libcephfs2-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:57.442 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:57.443 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:15:57.443 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:57.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs-devel 2024-04-26T04:15:57.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-04-26T04:15:58.034 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:41 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:58.042 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:40 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:58.114 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:58.155 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:58.156 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:58.156 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:58.163 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 32 k 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:15:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:58.165 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 32 k 2024-04-26T04:15:58.165 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 142 k 2024-04-26T04:15:58.165 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:15:58.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-04-26T04:15:58.406 INFO:teuthology.orchestra.run.smithi165.stdout:libcephfs-devel-19.0.0-2457.gddeaa917.el9.x86_6 131 kB/s | 32 kB 00:00 2024-04-26T04:15:58.407 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:15:58.407 INFO:teuthology.orchestra.run.smithi165.stdout:Total 131 kB/s | 32 kB 00:00 2024-04-26T04:15:58.407 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:15:58.414 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:15:58.414 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:15:58.414 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:15:58.414 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:15:58.497 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:15:58.602 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:58.731 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:41 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:58.810 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:58.850 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:58.851 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:58.851 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:58.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-04-26T04:15:58.964 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:15:59.172 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:59.172 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:15:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:15:59.173 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:15:59.173 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:15:59.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados2 2024-04-26T04:15:59.430 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:42 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:15:59.508 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:59.549 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:15:59.550 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:15:59.550 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:15:59.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-04-26T04:15:59.850 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:41 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:15:59.932 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:15:59.973 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:15:59.974 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:15:59.974 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:00.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librbd1 2024-04-26T04:16:00.128 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:00.206 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:00.246 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:00.247 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:16:00.247 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:00.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-04-26T04:16:00.551 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:42 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:00.631 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:00.672 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:00.674 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:16:00.674 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:00.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rados 2024-04-26T04:16:00.829 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:00.908 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:00.950 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:00.951 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:16:00.951 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:01.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-04-26T04:16:01.257 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:01.337 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rados-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:01.378 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:01.380 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:16:01.380 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:01.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rgw 2024-04-26T04:16:01.537 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:44 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:01.616 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:01.657 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:01.658 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-26T04:16:01.658 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:01.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-04-26T04:16:01.970 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:02.049 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rgw-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:02.089 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:02.090 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:16:02.090 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:02.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-cephfs 2024-04-26T04:16:02.238 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:45 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:02.358 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-26T04:16:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:02.359 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 86 k 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 86 k 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-04-26T04:16:02.360 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:16:02.678 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-2457.gddeaa917.el9.x86_64.rpm 269 kB/s | 86 kB 00:00 2024-04-26T04:16:02.678 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:02.679 INFO:teuthology.orchestra.run.smithi098.stdout:Total 268 kB/s | 86 kB 00:00 2024-04-26T04:16:02.679 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:16:02.686 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:16:02.686 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:16:02.686 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:44 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:02.748 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:16:02.748 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:16:02.766 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-cephfs-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:02.806 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:02.807 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:16:02.807 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:02.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rbd 2024-04-26T04:16:02.889 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:16:03.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:03.366 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:03.394 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:45 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:03.475 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rbd-2:19.0.0-2457.gddeaa917.el9.x86_64 is already installed. 2024-04-26T04:16:03.516 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:03.517 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-26T04:16:03.517 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:03.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-fuse 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:03.599 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:03.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-04-26T04:16:04.098 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:46 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:04.218 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:04.218 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 86 k 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 86 k 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 231 k 2024-04-26T04:16:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:16:04.385 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:47 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:04.434 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-fuse-19.0.0-2457.gddeaa917.el9.x86_64.rpm 398 kB/s | 86 kB 00:00 2024-04-26T04:16:04.435 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:04.435 INFO:teuthology.orchestra.run.smithi165.stdout:Total 396 kB/s | 86 kB 00:00 2024-04-26T04:16:04.435 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:16:04.442 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:16:04.442 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:16:04.504 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:16:04.505 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:16:04.508 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.1 M 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:04.509 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-04-26T04:16:04.510 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:16:04.650 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:16:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:05.323 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:05.323 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:16:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:05.324 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:05.324 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:05.441 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-2457.gddeaa917.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-04-26T04:16:05.442 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:05.442 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-04-26T04:16:05.442 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:16:05.451 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:16:05.451 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:16:05.531 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:16:05.531 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:16:05.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-mirror 2024-04-26T04:16:05.958 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:16:06.016 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T04:16:06.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:06.100 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:48 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:06.234 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:06.235 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:06.235 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-04-26T04:16:06.235 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 3.1 M 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.1 M 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 13 M 2024-04-26T04:16:06.236 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:16:06.806 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-mirror-19.0.0-2457.gddeaa917.el9.x86_64.rpm 5.4 MB/s | 3.1 MB 00:00 2024-04-26T04:16:06.806 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:06.806 INFO:teuthology.orchestra.run.smithi165.stdout:Total 5.4 MB/s | 3.1 MB 00:00 2024-04-26T04:16:06.806 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:16:06.814 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:16:06.814 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:16:06.895 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:16:06.895 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:16:07.040 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:07.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:07.040 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:16:07.041 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:07.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:07.041 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:07.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-04-26T04:16:07.319 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:16:07.371 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:07.396 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:07.396 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T04:16:07.396 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T04:16:07.397 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T04:16:07.397 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T04:16:07.397 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:07.777 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:50 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:16:07.899 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-26T04:16:07.900 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:07.900 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-26T04:16:07.900 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:07.900 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-26T04:16:07.900 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 170 k 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 170 k 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-04-26T04:16:07.901 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-26T04:16:08.121 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-2457.gddeaa917.el9.x86_64.rpm 777 kB/s | 170 kB 00:00 2024-04-26T04:16:08.122 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:08.122 INFO:teuthology.orchestra.run.smithi098.stdout:Total 771 kB/s | 170 kB 00:00 2024-04-26T04:16:08.122 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-26T04:16:08.129 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:16:08.129 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-26T04:16:08.193 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:16:08.193 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-26T04:16:08.351 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-26T04:16:08.466 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:08.479 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:08.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-nbd 2024-04-26T04:16:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:09.149 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:09.150 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-26T04:16:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:09.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:09.150 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-26T04:16:09.202 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:51 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:16:09.327 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.0.0-2457.gddeaa917.el9 ceph 170 k 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:09.328 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 170 k 2024-04-26T04:16:09.329 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 494 k 2024-04-26T04:16:09.329 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-26T04:16:09.391 DEBUG:teuthology.parallel:result is None 2024-04-26T04:16:09.690 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-nbd-19.0.0-2457.gddeaa917.el9.x86_64.rpm 472 kB/s | 170 kB 00:00 2024-04-26T04:16:09.691 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:09.691 INFO:teuthology.orchestra.run.smithi165.stdout:Total 469 kB/s | 170 kB 00:00 2024-04-26T04:16:09.691 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-26T04:16:09.699 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:16:09.699 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-26T04:16:09.764 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:16:09.764 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-26T04:16:09.920 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-26T04:16:10.019 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:10.442 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 1/1 2024-04-26T04:16:10.695 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:10.696 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-04-26T04:16:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.0.0-2457.gddeaa917.el9.x86_64 2024-04-26T04:16:10.696 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:16:10.696 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-26T04:16:10.952 DEBUG:teuthology.parallel:result is None 2024-04-26T04:16:10.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:11.105 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T04:16:11.129 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2457.gddeaa917.el9 2024-04-26T04:16:11.130 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457.gddeaa917.el9 2024-04-26T04:16:11.130 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457.gddeaa917 is installed. 2024-04-26T04:16:11.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:11.387 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T04:16:11.409 INFO:teuthology.orchestra.run.smithi165.stdout:19.0.0-2457.gddeaa917.el9 2024-04-26T04:16:11.409 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457.gddeaa917.el9 2024-04-26T04:16:11.409 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457.gddeaa917 is installed. 2024-04-26T04:16:11.411 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T04:16:11.411 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:16:11.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T04:16:11.440 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:16:11.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T04:16:11.473 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T04:16:11.473 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:16:11.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T04:16:11.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T04:16:11.572 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:16:11.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T04:16:11.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T04:16:11.660 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T04:16:11.660 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:16:11.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T04:16:11.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T04:16:11.750 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:16:11.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T04:16:11.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T04:16:11.837 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T04:16:11.837 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:16:11.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T04:16:11.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T04:16:11.924 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:16:11.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T04:16:11.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T04:16:12.011 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T04:16:12.112 INFO:tasks.cephadm:Config: {'conf': {'client': {'debug client': 20, 'debug mgrc': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 1}, 'mds': {'debug mgrc': 20}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug mgrc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'} 2024-04-26T04:16:12.112 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:12.113 INFO:tasks.cephadm:Cluster fsid is b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:16:12.113 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T04:16:12.113 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98'} 2024-04-26T04:16:12.113 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-04-26T04:16:12.113 INFO:tasks.cephadm:First mgr is a 2024-04-26T04:16:12.113 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T04:16:12.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-04-26T04:16:12.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-04-26T04:16:12.162 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T04:16:12.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:12.302 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78858/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2457-gddeaa917', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2457.gddeaa917'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 19:22:02.305725', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri2-testing-2024-04-24-0914-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T04:16:12.680 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-yuri2-testing-2024-04-24-0914-squid, sha1 ddeaa91797339a19172c3036ff48cca58c12f448 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:12.681 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/x86_64/flavors/default/cephadm 2024-04-26T04:16:12.681 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:13.896 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 781341 Apr 26 04:16 /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:13.897 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:14.469 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 781341 Apr 26 04:16 /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:14.469 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:14.488 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T04:16:14.517 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 on all hosts... 2024-04-26T04:16:14.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 pull 2024-04-26T04:16:14.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 pull 2024-04-26T04:16:16.009 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448... 2024-04-26T04:16:16.074 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448... 2024-04-26T04:16:49.059 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2457-gddeaa917 (ddeaa91797339a19172c3036ff48cca58c12f448) squid (dev)", 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4" 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-26T04:16:49.060 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.0.0-2457-gddeaa917 (ddeaa91797339a19172c3036ff48cca58c12f448) squid (dev)", 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4" 2024-04-26T04:16:51.707 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-04-26T04:16:51.708 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-26T04:16:51.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-04-26T04:16:51.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-04-26T04:16:51.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-04-26T04:16:51.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-04-26T04:16:51.856 INFO:tasks.cephadm:Writing seed config... 2024-04-26T04:16:51.857 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T04:16:51.857 INFO:tasks.cephadm: override: [client] debug mgrc = 20 2024-04-26T04:16:51.857 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T04:16:51.857 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mds] debug mgrc = 20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] bluestore compression algorithm = zlib 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-26T04:16:51.858 INFO:tasks.cephadm: override: [osd] debug mgrc = 20 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T04:16:51.859 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T04:16:51.860 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:16:51.860 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T04:16:51.890 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b735c86e-0383-11ef-bc93-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression algorithm = zlib bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug mgrc = 20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug client = 20 debug mgrc = 20 debug ms = 1 [mds] debug mgrc = 20 2024-04-26T04:16:51.891 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T04:16:51.933 DEBUG:teuthology.orchestra.run.smithi098:mgr.a> sudo journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service 2024-04-26T04:16:51.976 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T04:16:51.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 -v bootstrap --fsid b735c86e-0383-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id a --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:16:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:16:52.169 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448', '-v', 'bootstrap', '--fsid', 'b735c86e-0383-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'a', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-04-26T04:16:52.169 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T04:16:52.170 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-04-26T04:16:52.195 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T04:16:52.195 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-04-26T04:16:52.195 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-04-26T04:16:52.202 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T04:16:52.202 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T04:16:52.208 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T04:16:52.208 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:16:52.214 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-26T04:16:52.220 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-26T04:16:52.220 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-26T04:16:52.220 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-04-26T04:16:52.244 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T04:16:52.244 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T04:16:52.244 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-04-26T04:16:52.244 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-04-26T04:16:52.250 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T04:16:52.250 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T04:16:52.256 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T04:16:52.256 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:16:52.262 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 140290865249824 on /run/cephadm/b735c86e-0383-11ef-bc93-c7b262605968.lock 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 140290865249824 acquired on /run/cephadm/b735c86e-0383-11ef-bc93-c7b262605968.lock 2024-04-26T04:16:52.269 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-04-26T04:16:52.270 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-04-26T04:16:52.270 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-26T04:16:52.273 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-04-26T04:16:52.273 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-04-26T04:16:52.275 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T04:16:52.275 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-26T04:16:52.275 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-26T04:16:52.277 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T04:16:52.277 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::47eb:1f36:84bd:bfb1/64 scope link noprefixroute 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-26T04:16:52.278 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T04:16:52.279 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T04:16:52.279 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448... 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448... 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:674f25154a7d2ad4928deef0be46d4116a4edd413dc953486fdeb3f13351ee37 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 2024-04-26T04:16:52.720 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T04:16:53.700 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.0.0-2457-gddeaa917 (ddeaa91797339a19172c3036ff48cca58c12f448) squid (dev) 2024-04-26T04:16:53.701 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.0.0-2457-gddeaa917 (ddeaa91797339a19172c3036ff48cca58c12f448) squid (dev) 2024-04-26T04:16:53.701 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T04:16:54.689 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-04-26T04:16:54.689 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-04-26T04:16:55.678 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC3KitmN87QARAARsQqq8hHgt4tfCO8Z8Pg2g== 2024-04-26T04:16:56.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC4Kitm3FBbARAA+tlowwvoRoSr0iaYxURL4g== 2024-04-26T04:16:57.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC5KitmBTCqAhAAjbF+CoiDHsjLsCVNno7fvw== 2024-04-26T04:16:57.578 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-04-26T04:16:58.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:16:58.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T04:16:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:16:58.504 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-04-26T04:16:59.383 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-04-26T04:16:59.588 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T04:16:59.726 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T04:16:59.951 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target. 2024-04-26T04:16:59.951 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target. 2024-04-26T04:17:00.195 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a 2024-04-26T04:17:00.196 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T04:17:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target.wants/ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968@.service. 2024-04-26T04:17:00.933 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T04:17:00.933 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-26T04:17:00.940 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T04:17:00.940 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:17:00.940 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-26T04:17:00.941 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T04:17:00.941 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-04-26T04:17:00.941 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-04-26T04:17:01.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:01 smithi098 ceph-mon[26814]: from='client.? 172.21.15.98:0/366008136' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.541542s) 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T04:17:02.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.352+0000 7f64248e2700 1 Processor -- start 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.352+0000 7f64248e2700 1 -- start start 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.352+0000 7f64248e2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.352+0000 7f64248e2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6420075630 con 0x7f6420105060 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53866/0 (socket says 172.21.15.98:53866) 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641e59c700 1 -- 172.21.15.98:0/1861448529 learned_addr learned my addr 172.21.15.98:0/1861448529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641e59c700 1 -- 172.21.15.98:0/1861448529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64201059c0 con 0x7f6420105060 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641e59c700 1 --2- 172.21.15.98:0/1861448529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64100098a0 tx=0x7f6410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66050e9e14197d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:02.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641d59a700 1 -- 172.21.15.98:0/1861448529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410004160 con 0x7f6420105060 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641d59a700 1 -- 172.21.15.98:0/1861448529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6410004300 con 0x7f6420105060 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.353+0000 7f641d59a700 1 -- 172.21.15.98:0/1861448529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64100036a0 con 0x7f6420105060 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 -- 172.21.15.98:0/1861448529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 msgr2=0x7f6420105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 --2- 172.21.15.98:0/1861448529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f64100098a0 tx=0x7f6410009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 -- 172.21.15.98:0/1861448529 shutdown_connections 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 --2- 172.21.15.98:0/1861448529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f6420105480 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 -- 172.21.15.98:0/1861448529 >> 172.21.15.98:0/1861448529 conn(0x7f6420100c60 msgr2=0x7f6420103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 -- 172.21.15.98:0/1861448529 shutdown_connections 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.354+0000 7f64248e2700 1 -- 172.21.15.98:0/1861448529 wait complete. 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f64248e2700 1 Processor -- start 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f64248e2700 1 -- start start 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f64248e2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f64248e2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6420105ef0 con 0x7f6420105060 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f641e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:02.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f641e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53872/0 (socket says 172.21.15.98:53872) 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f641e59c700 1 -- 172.21.15.98:0/366008136 learned_addr learned my addr 172.21.15.98:0/366008136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.355+0000 7f641e59c700 1 -- 172.21.15.98:0/366008136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642019c680 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f641e59c700 1 --2- 172.21.15.98:0/366008136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f6410004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410003db0 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6410011630 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642019c910 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642019d430 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6410011950 con 0x7f6420105060 2024-04-26T04:17:02.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f641001a460 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.356+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f641001b780 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.357+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6420063850 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.359+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6410031020 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.434+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f642019d6b0 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.435+0000 7f640f7fe700 1 -- 172.21.15.98:0/366008136 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f641001b9d0 con 0x7f6420105060 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.435+0000 7f64248e2700 10 mgrc shutdown 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.435+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 msgr2=0x7f642019c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 --2- 172.21.15.98:0/366008136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f6410004750 comp rx=0 tx=0).stop 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 shutdown_connections 2024-04-26T04:17:02.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 --2- 172.21.15.98:0/366008136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6420105060 0x7f642019c140 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 >> 172.21.15.98:0/366008136 conn(0x7f6420100c60 msgr2=0x7f6420192680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 shutdown_connections 2024-04-26T04:17:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:01.436+0000 7f64248e2700 1 -- 172.21.15.98:0/366008136 wait complete. 2024-04-26T04:17:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-04-26T04:17:02.006 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T04:17:03.107 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:02 smithi098 ceph-mon[26814]: from='client.? 172.21.15.98:0/327532531' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T04:17:03.107 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:02 smithi098 ceph-mon[26814]: from='client.? 172.21.15.98:0/327532531' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T04:17:03.116 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:03.116 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T04:17:03.117 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.429+0000 7f0b95da1700 1 Processor -- start 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.429+0000 7f0b95da1700 1 -- start start 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.429+0000 7f0b95da1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.429+0000 7f0b95da1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90106590 con 0x7f0b90105ba0 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53888/0 (socket says 172.21.15.98:53888) 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/495303190 learned_addr learned my addr 172.21.15.98:0/495303190 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/495303190 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b901068c0 con 0x7f0b90105ba0 2024-04-26T04:17:03.118 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/495303190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098a0 tx=0x7f0b80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b952a337db6ba179 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8e7fc700 1 -- 172.21.15.98:0/495303190 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b80004160 con 0x7f0b90105ba0 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8e7fc700 1 -- 172.21.15.98:0/495303190 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b80004300 con 0x7f0b90105ba0 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.430+0000 7f0b8e7fc700 1 -- 172.21.15.98:0/495303190 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b800036a0 con 0x7f0b90105ba0 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- 172.21.15.98:0/495303190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 msgr2=0x7f0b90105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 --2- 172.21.15.98:0/495303190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098a0 tx=0x7f0b80009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- 172.21.15.98:0/495303190 shutdown_connections 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 --2- 172.21.15.98:0/495303190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90105fc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- 172.21.15.98:0/495303190 >> 172.21.15.98:0/495303190 conn(0x7f0b90100400 msgr2=0x7f0b90102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- 172.21.15.98:0/495303190 shutdown_connections 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- 172.21.15.98:0/495303190 wait complete. 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 Processor -- start 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.431+0000 7f0b95da1700 1 -- start start 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b95da1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:03.119 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b95da1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90106df0 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53892/0 (socket says 172.21.15.98:53892) 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/327532531 learned_addr learned my addr 172.21.15.98:0/327532531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/327532531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b90198110 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/327532531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b80000c00 tx=0x7f0b80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b80003db0 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b80011630 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b901983a0 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b80011950 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.432+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b9019b090 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b80011af0 con 0x7f0b90105ba0 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:03.120 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0b8001c450 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.433+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b90063850 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.435+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0b80032020 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.503+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0b901907c0 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.506+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f0b80004260 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.506+0000 7f0b8cff9700 1 -- 172.21.15.98:0/327532531 <== mon.0 v2:172.21.15.98:3300/0 8 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f0b8001c650 con 0x7f0b90105ba0 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 10 mgrc shutdown 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 msgr2=0x7f0b90197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 --2- 172.21.15.98:0/327532531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b80000c00 tx=0x7f0b80004750 comp rx=0 tx=0).stop 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 shutdown_connections 2024-04-26T04:17:03.121 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 --2- 172.21.15.98:0/327532531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90105ba0 0x7f0b90197bd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 >> 172.21.15.98:0/327532531 conn(0x7f0b90100400 msgr2=0x7f0b9018e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 shutdown_connections 2024-04-26T04:17:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:02.507+0000 7f0b95da1700 1 -- 172.21.15.98:0/327532531 wait complete. 2024-04-26T04:17:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-04-26T04:17:04.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:03 smithi098 ceph-mon[26814]: from='client.? 172.21.15.98:0/131361193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffab08aa700 1 Processor -- start 2024-04-26T04:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffab08aa700 1 -- start start 2024-04-26T04:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffab08aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffab08aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac079e00 con 0x7ffaac105060 2024-04-26T04:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffaaa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffaaa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53906/0 (socket says 172.21.15.98:53906) 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.543+0000 7ffaaa59c700 1 -- 172.21.15.98:0/980931980 learned_addr learned my addr 172.21.15.98:0/980931980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.544+0000 7ffaaa59c700 1 -- 172.21.15.98:0/980931980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac1059c0 con 0x7ffaac105060 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.544+0000 7ffaaa59c700 1 --2- 172.21.15.98:0/980931980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0098a0 tx=0x7ffa9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be33108b9c3d3cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.544+0000 7ffaa959a700 1 -- 172.21.15.98:0/980931980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa9c004160 con 0x7ffaac105060 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.544+0000 7ffaa959a700 1 -- 172.21.15.98:0/980931980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7ffa9c004300 con 0x7ffaac105060 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.544+0000 7ffaa959a700 1 -- 172.21.15.98:0/980931980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa9c0119b0 con 0x7ffaac105060 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 -- 172.21.15.98:0/980931980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 msgr2=0x7ffaac105480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 --2- 172.21.15.98:0/980931980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0098a0 tx=0x7ffa9c009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 -- 172.21.15.98:0/980931980 shutdown_connections 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 --2- 172.21.15.98:0/980931980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac105480 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:04.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 -- 172.21.15.98:0/980931980 >> 172.21.15.98:0/980931980 conn(0x7ffaac100c60 msgr2=0x7ffaac103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 -- 172.21.15.98:0/980931980 shutdown_connections 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 -- 172.21.15.98:0/980931980 wait complete. 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.545+0000 7ffab08aa700 1 Processor -- start 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffab08aa700 1 -- start start 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffab08aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffab08aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac106990 con 0x7ffaac105060 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffaaa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffaaa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53914/0 (socket says 172.21.15.98:53914) 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffaaa59c700 1 -- 172.21.15.98:0/131361193 learned_addr learned my addr 172.21.15.98:0/131361193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffaaa59c700 1 -- 172.21.15.98:0/131361193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac19cd20 con 0x7ffaac105060 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffaaa59c700 1 --2- 172.21.15.98:0/131361193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c011670 tx=0x7ffa9c011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa9c0038b0 con 0x7ffaac105060 2024-04-26T04:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac19cfb0 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7ffa9c01a8a0 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa9c019550 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.546+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac19fca0 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa9c019c00 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffa9c024730 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.547+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac063850 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.550+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ffa9c043dd0 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.617+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ffaac02db90 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.617+0000 7ffa9b7fe700 1 -- 172.21.15.98:0/131361193 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7ffa9c032020 con 0x7ffaac105060 2024-04-26T04:17:04.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 10 mgrc shutdown 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 msgr2=0x7ffaac19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 --2- 172.21.15.98:0/131361193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c011670 tx=0x7ffa9c011820 comp rx=0 tx=0).stop 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 shutdown_connections 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 --2- 172.21.15.98:0/131361193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffaac105060 0x7ffaac19c7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 >> 172.21.15.98:0/131361193 conn(0x7ffaac100c60 msgr2=0x7ffaac193480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 shutdown_connections 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:03.618+0000 7ffab08aa700 1 -- 172.21.15.98:0/131361193 wait complete. 2024-04-26T04:17:04.193 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-04-26T04:17:04.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:04 smithi098 systemd[1]: Stopping Ceph mon.a for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:17:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:04 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a[26808]: 2024-04-26T04:17:04.457+0000 7fe9e8331700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:17:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:04 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a[26808]: 2024-04-26T04:17:04.457+0000 7fe9e8331700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T04:17:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:04 smithi098 podman[27058]: 2024-04-26 04:17:04.672940508 +0000 UTC m=+0.322565125 container died e240eb54b8908a9692f5f4109ce5df459d86983d3dd707ceb2db3c617e4e5ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T04:17:05.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27058]: 2024-04-26 04:17:05.041091687 +0000 UTC m=+0.690716307 container cleanup e240eb54b8908a9692f5f4109ce5df459d86983d3dd707ceb2db3c617e4e5ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T04:17:05.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 bash[27058]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a 2024-04-26T04:17:05.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27070]: 2024-04-26 04:17:05.291960287 +0000 UTC m=+0.615128963 container remove e240eb54b8908a9692f5f4109ce5df459d86983d3dd707ceb2db3c617e4e5ee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-26T04:17:05.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 systemd[1]: ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T04:17:05.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 systemd[1]: Stopped Ceph mon.a for b735c86e-0383-11ef-bc93-c7b262605968. 2024-04-26T04:17:05.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 systemd[1]: Starting Ceph mon.a for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:17:05.920 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27165]: 2024-04-26 04:17:05.634503418 +0000 UTC m=+0.123052757 container create a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27165]: 2024-04-26 04:17:05.539304662 +0000 UTC m=+0.027854002 image pull b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27165]: 2024-04-26 04:17:05.828409069 +0000 UTC m=+0.316958414 container init a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 podman[27165]: 2024-04-26 04:17:05.831381565 +0000 UTC m=+0.319930911 container start a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: ceph version 19.0.0-2457-gddeaa917 (ddeaa91797339a19172c3036ff48cca58c12f448) squid (dev), process ceph-mon, pid 2 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: pidfile_write: ignore empty --pid-file 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: load: jerasure load: lrc 2024-04-26T04:17:05.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: RocksDB version: 7.9.2 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Git sha 0 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Compile date 2024-04-24 18:24:05 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: DB SUMMARY 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: DB Session ID: JIW7KDKC4LB9RT1OZ473 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: CURRENT file: CURRENT 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95605 ; 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.error_if_exists: 0 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.create_if_missing: 0 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.paranoid_checks: 1 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T04:17:05.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.env: 0x55d3d928ac60 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.info_log: 0x55d3da9b3a80 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.statistics: (nil) 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.use_fsync: 0 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_log_file_size: 0 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_fallocate: 1 2024-04-26T04:17:05.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.use_direct_reads: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.db_log_dir: 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.wal_dir: 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T04:17:05.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.write_buffer_manager: 0x55d3da9861e0 2024-04-26T04:17:05.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T04:17:05.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T04:17:05.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T04:17:05.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T04:17:05.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.unordered_write: 0 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T04:17:05.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.row_cache: None 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.wal_filter: None 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.two_write_queues: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.wal_compression: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.atomic_flush: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.log_readahead_size: 0 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T04:17:05.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_background_jobs: 2 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_background_compactions: -1 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_subcompactions: 1 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T04:17:05.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_open_files: -1 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_background_flushes: -1 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Compression algorithms supported: 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kZSTD supported: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kXpressCompression supported: 0 2024-04-26T04:17:05.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kZlibCompression supported: 1 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kSnappyCompression supported: 1 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kLZ4Compression supported: 1 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: kBZip2Compression supported: 0 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.merge_operator: 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_filter: None 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T04:17:05.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d3da9b3b80) 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x55d3da9b7090 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-04-26T04:17:05.932 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-04-26T04:17:05.933 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-04-26T04:17:05.933 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-04-26T04:17:05.933 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-04-26T04:17:05.933 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-04-26T04:17:05.933 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-04-26T04:17:05.934 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression: NoCompression 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T04:17:05.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.num_levels: 7 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T04:17:05.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T04:17:05.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T04:17:05.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T04:17:05.939 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T04:17:05.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.table_properties_collectors: 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.inplace_update_support: 0 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T04:17:05.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.bloom_locality: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.max_successive_merges: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T04:17:05.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.ttl: 2592000 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enable_blob_files: false 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.min_blob_size: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T04:17:05.943 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ccb16d6d-3e60-45b0-b983-3152d228f97a 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714105025859839, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714105025860797, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91521, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 288, "table_properties": {"data_size": 89588, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11884, "raw_average_key_size": 48, "raw_value_size": 83043, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 247, "num_filter_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714105025, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ccb16d6d-3e60-45b0-b983-3152d228f97a", "db_session_id": "JIW7KDKC4LB9RT1OZ473", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714105025860882, "job": 1, "event": "recovery_finished"} 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d3daaa8000 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: DB pointer 0x55d3daa92000 2024-04-26T04:17:05.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T04:17:05.945 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 91.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 91.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.946 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 30.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x55d3da9b7090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-26T04:17:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???) e1 preinit fsid b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).mds e1 new map 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).mds e1 print_map 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T04:17:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 bash[27165]: a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 systemd[1]: Started Ceph mon.a for b735c86e-0383-11ef-bc93-c7b262605968. 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: monmap epoch 1 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: fsid b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: last_changed 2024-04-26T04:16:57.901441+0000 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: created 2024-04-26T04:16:57.901441+0000 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: min_mon_release 19 (squid) 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: election_strategy: 1 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: fsmap 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T04:17:05.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:05 smithi098 ceph-mon[27184]: mgrmap e1: no daemons active 2024-04-26T04:17:05.950 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.332+0000 7f11a2ef4700 1 Processor -- start 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.332+0000 7f11a2ef4700 1 -- start start 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a2ef4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a2ef4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f119c106390 con 0x7f119c107e50 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a0c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:07.027 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a0c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53920/0 (socket says 172.21.15.98:53920) 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a0c90700 1 -- 172.21.15.98:0/2955399617 learned_addr learned my addr 172.21.15.98:0/2955399617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a0c90700 1 -- 172.21.15.98:0/2955399617 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c106510 con 0x7f119c107e50 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.333+0000 7f11a0c90700 1 --2- 172.21.15.98:0/2955399617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f11900098a0 tx=0x7f1190009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3954acbeead5dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f119b7fe700 1 -- 172.21.15.98:0/2955399617 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1190004160 con 0x7f119c107e50 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f119b7fe700 1 -- 172.21.15.98:0/2955399617 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f1190004300 con 0x7f119c107e50 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f119b7fe700 1 -- 172.21.15.98:0/2955399617 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11900119b0 con 0x7f119c107e50 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 -- 172.21.15.98:0/2955399617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 msgr2=0x7f119c108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 --2- 172.21.15.98:0/2955399617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f11900098a0 tx=0x7f1190009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 -- 172.21.15.98:0/2955399617 shutdown_connections 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 --2- 172.21.15.98:0/2955399617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c108270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 -- 172.21.15.98:0/2955399617 >> 172.21.15.98:0/2955399617 conn(0x7f119c101cf0 msgr2=0x7f119c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 -- 172.21.15.98:0/2955399617 shutdown_connections 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.334+0000 7f11a2ef4700 1 -- 172.21.15.98:0/2955399617 wait complete. 2024-04-26T04:17:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a2ef4700 1 Processor -- start 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a2ef4700 1 -- start start 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a2ef4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a2ef4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10ad20 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a0c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a0c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53936/0 (socket says 172.21.15.98:53936) 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a0c90700 1 -- 172.21.15.98:0/1184437805 learned_addr learned my addr 172.21.15.98:0/1184437805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.335+0000 7f11a0c90700 1 -- 172.21.15.98:0/1184437805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c19cec0 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f11a0c90700 1 --2- 172.21.15.98:0/1184437805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1190011670 tx=0x7f1190011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11900036a0 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f1190003840 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1190019580 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c19d150 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c19fe40 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f119002d440 con 0x7f119c107e50 2024-04-26T04:17:07.029 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.336+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1190024820 con 0x7f119c107e50 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.337+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c10aea0 con 0x7f119c107e50 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.339+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1190043dd0 con 0x7f119c107e50 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.405+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f119c1a00f0 con 0x7f119c107e50 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f1199ffb700 1 -- 172.21.15.98:0/1184437805 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f1190032020 con 0x7f119c107e50 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 10 mgrc shutdown 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 msgr2=0x7f119c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 1 --2- 172.21.15.98:0/1184437805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1190011670 tx=0x7f1190011820 comp rx=0 tx=0).stop 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 shutdown_connections 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 1 --2- 172.21.15.98:0/1184437805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f119c107e50 0x7f119c19c980 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:07.030 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.410+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 >> 172.21.15.98:0/1184437805 conn(0x7f119c101cf0 msgr2=0x7f119c102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:07.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.411+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 shutdown_connections 2024-04-26T04:17:07.031 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:06.411+0000 7f11a2ef4700 1 -- 172.21.15.98:0/1184437805 wait complete. 2024-04-26T04:17:07.031 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T04:17:07.043 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:17:07.043 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-04-26T04:17:07.044 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T04:17:07.044 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T04:17:07.269 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a 2024-04-26T04:17:07.269 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service: Unit ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service not loaded. 2024-04-26T04:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968.target.wants/ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service → /etc/systemd/system/ceph-b735c86e-0383-11ef-bc93-c7b262605968@.service. 2024-04-26T04:17:07.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:07 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1184437805' entity='client.admin' 2024-04-26T04:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T04:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-26T04:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T04:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-26T04:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T04:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T04:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-26T04:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T04:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-26T04:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T04:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-04-26T04:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-04-26T04:17:08.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:08 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1397232720' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T04:17:08.685 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:08 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:08.600+0000 7f6c22df1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T04:17:08.686 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:08 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:08.683+0000 7f6c22df1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T04:17:09.087 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "b735c86e-0383-11ef-bc93-c7b262605968", 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T04:17:09.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T04:17:09.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:09.090 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:09.091 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T04:17:00.896134+0000", 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50e21b5700 1 Processor -- start 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50e21b5700 1 -- start start 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50e21b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50e21b5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc106070 con 0x7f50dc105680 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53960/0 (socket says 172.21.15.98:53960) 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.342+0000 7f50db7fe700 1 -- 172.21.15.98:0/3708366975 learned_addr learned my addr 172.21.15.98:0/3708366975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.343+0000 7f50db7fe700 1 -- 172.21.15.98:0/3708366975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc106880 con 0x7f50dc105680 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.343+0000 7f50db7fe700 1 --2- 172.21.15.98:0/3708366975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153e6f82171c1905 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:09.092 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.343+0000 7f50da7fc700 1 -- 172.21.15.98:0/3708366975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc004160 con 0x7f50dc105680 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.343+0000 7f50da7fc700 1 -- 172.21.15.98:0/3708366975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f50cc004300 con 0x7f50dc105680 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.343+0000 7f50da7fc700 1 -- 172.21.15.98:0/3708366975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc0119b0 con 0x7f50dc105680 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 -- 172.21.15.98:0/3708366975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 msgr2=0x7f50dc105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 --2- 172.21.15.98:0/3708366975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 -- 172.21.15.98:0/3708366975 shutdown_connections 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 --2- 172.21.15.98:0/3708366975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc105aa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 -- 172.21.15.98:0/3708366975 >> 172.21.15.98:0/3708366975 conn(0x7f50dc100c60 msgr2=0x7f50dc103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 -- 172.21.15.98:0/3708366975 shutdown_connections 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 -- 172.21.15.98:0/3708366975 wait complete. 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.344+0000 7f50e21b5700 1 Processor -- start 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50e21b5700 1 -- start start 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50e21b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50e21b5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0747d0 con 0x7f50dc105680 2024-04-26T04:17:09.093 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53974/0 (socket says 172.21.15.98:53974) 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50db7fe700 1 -- 172.21.15.98:0/1397232720 learned_addr learned my addr 172.21.15.98:0/1397232720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50db7fe700 1 -- 172.21.15.98:0/1397232720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc198a80 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50db7fe700 1 --2- 172.21.15.98:0/1397232720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006af0 tx=0x7f50cc003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc0038b0 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f50cc01a8a0 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc0246e0 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.345+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc198d10 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc199830 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f50cc022020 con 0x7f50dc105680 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:09.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f50cc02dc90 con 0x7f50dc105680 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.346+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc063850 con 0x7f50dc105680 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.349+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f50cc043dd0 con 0x7f50dc105680 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.419+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f50dc19a280 con 0x7f50dc105680 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.420+0000 7f50d8ff9700 1 -- 172.21.15.98:0/1397232720 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f50cc032020 con 0x7f50dc105680 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 10 mgrc shutdown 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 msgr2=0x7f50dc198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 --2- 172.21.15.98:0/1397232720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006af0 tx=0x7f50cc003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 shutdown_connections 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 --2- 172.21.15.98:0/1397232720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50dc105680 0x7f50dc198540 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 >> 172.21.15.98:0/1397232720 conn(0x7f50dc100c60 msgr2=0x7f50dc18eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 shutdown_connections 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:08.421+0000 7f50e21b5700 1 -- 172.21.15.98:0/1397232720 wait complete. 2024-04-26T04:17:09.095 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-04-26T04:17:10.184 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:09 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:09.853+0000 7f6c22df1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T04:17:10.184 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.018+0000 7f6c22df1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:10.185 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.182+0000 7f6c22df1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T04:17:10.749 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.461+0000 7f6c22df1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T04:17:10.749 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.550+0000 7f6c22df1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T04:17:10.749 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.747+0000 7f6c22df1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T04:17:11.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:10 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:10.948+0000 7f6c22df1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:17:11.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:11 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/4204758442' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T04:17:11.791 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:11 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:11.556+0000 7f6c22df1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T04:17:12.271 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:12 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:11.998+0000 7f6c22df1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T04:17:12.271 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:12 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:12.093+0000 7f6c22df1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T04:17:12.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:12.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:12.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "b735c86e-0383-11ef-bc93-c7b262605968", 2024-04-26T04:17:12.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-26T04:17:12.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T04:17:12.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T04:17:12.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T04:17:00.896134+0000", 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:12.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f5498f77700 1 Processor -- start 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f5498f77700 1 -- start start 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f5498f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f5498f77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5494106070 con 0x7f5494105680 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f549259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f549259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60606/0 (socket says 172.21.15.98:60606) 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.544+0000 7f549259c700 1 -- 172.21.15.98:0/1027897026 learned_addr learned my addr 172.21.15.98:0/1027897026 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.545+0000 7f549259c700 1 -- 172.21.15.98:0/1027897026 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5494106880 con 0x7f5494105680 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.545+0000 7f549259c700 1 --2- 172.21.15.98:0/1027897026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54880098a0 tx=0x7f5488009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51de3d3d639586da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.545+0000 7f549159a700 1 -- 172.21.15.98:0/1027897026 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5488004160 con 0x7f5494105680 2024-04-26T04:17:12.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.545+0000 7f549159a700 1 -- 172.21.15.98:0/1027897026 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f5488004300 con 0x7f5494105680 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.545+0000 7f549159a700 1 -- 172.21.15.98:0/1027897026 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54880119b0 con 0x7f5494105680 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 -- 172.21.15.98:0/1027897026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 msgr2=0x7f5494105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 --2- 172.21.15.98:0/1027897026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54880098a0 tx=0x7f5488009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 -- 172.21.15.98:0/1027897026 shutdown_connections 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 --2- 172.21.15.98:0/1027897026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494105aa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 -- 172.21.15.98:0/1027897026 >> 172.21.15.98:0/1027897026 conn(0x7f5494100c60 msgr2=0x7f5494103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 -- 172.21.15.98:0/1027897026 shutdown_connections 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.546+0000 7f5498f77700 1 -- 172.21.15.98:0/1027897026 wait complete. 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f5498f77700 1 Processor -- start 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f5498f77700 1 -- start start 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f5498f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f5498f77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5494109ab0 con 0x7f5494105680 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f549259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:12.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f549259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60608/0 (socket says 172.21.15.98:60608) 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f549259c700 1 -- 172.21.15.98:0/4204758442 learned_addr learned my addr 172.21.15.98:0/4204758442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f549259c700 1 -- 172.21.15.98:0/4204758442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5494199bb0 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.547+0000 7f549259c700 1 --2- 172.21.15.98:0/4204758442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5488006af0 tx=0x7f5488003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54880038b0 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f548801a8a0 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5494199e40 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54880246e0 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549419a960 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5488022020 con 0x7f5494105680 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 20 mgrc handle_mgr_map mgrmap(e 1) 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 4 mgrc handle_mgr_map Got map version 1 2024-04-26T04:17:12.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.548+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5488019490 con 0x7f5494105680 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.549+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5494063850 con 0x7f5494105680 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.551+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5488043dd0 con 0x7f5494105680 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.641+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f549419abe0 con 0x7f5494105680 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.642+0000 7f54837fe700 1 -- 172.21.15.98:0/4204758442 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5488032020 con 0x7f5494105680 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 10 mgrc shutdown 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 msgr2=0x7f5494199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 --2- 172.21.15.98:0/4204758442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5488006af0 tx=0x7f5488003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 shutdown_connections 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 --2- 172.21.15.98:0/4204758442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5494105680 0x7f5494199670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 >> 172.21.15.98:0/4204758442 conn(0x7f5494100c60 msgr2=0x7f54941017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:12.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 shutdown_connections 2024-04-26T04:17:12.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:11.643+0000 7f5498f77700 1 -- 172.21.15.98:0/4204758442 wait complete. 2024-04-26T04:17:12.296 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-04-26T04:17:12.541 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:12 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:12.293+0000 7f6c22df1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T04:17:13.128 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:12 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:12.803+0000 7f6c22df1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T04:17:13.129 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:12 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:12.892+0000 7f6c22df1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T04:17:13.464 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:13 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:13.126+0000 7f6c22df1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T04:17:13.464 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:13 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:13.257+0000 7f6c22df1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T04:17:13.464 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:13 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:13.350+0000 7f6c22df1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T04:17:13.464 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:13 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:13.462+0000 7f6c22df1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:13.791 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:13 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:13.553+0000 7f6c22df1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T04:17:14.651 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:14.382+0000 7f6c22df1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T04:17:14.651 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:14.593+0000 7f6c22df1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: Activating manager daemon a 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: mgrmap e2: a(active, starting, since 0.00299069s) 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix": "mgr metadata", "who": "a", "id": "a"}]: dispatch 2024-04-26T04:17:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: Manager daemon a is now available 2024-04-26T04:17:15.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/mirror_snapshot_schedule"}]: dispatch 2024-04-26T04:17:15.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/trash_purge_schedule"}]: dispatch 2024-04-26T04:17:15.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' 2024-04-26T04:17:15.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' 2024-04-26T04:17:15.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:14 smithi098 ceph-mon[27184]: from='mgr.14100 172.21.15.98:0/4109508857' entity='mgr.a' 2024-04-26T04:17:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:16 smithi098 ceph-mon[27184]: mgrmap e3: a(active, since 1.0068s) 2024-04-26T04:17:17.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:17 smithi098 ceph-mon[27184]: mgrmap e4: a(active, since 2s) 2024-04-26T04:17:18.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:18 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3386542152' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "b735c86e-0383-11ef-bc93-c7b262605968", 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T04:17:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T04:17:24.585 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T04:17:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T04:17:00.896134+0000", 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T04:17:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.581+0000 7f79d4f8a700 1 Processor -- start 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.581+0000 7f79d4f8a700 1 -- start start 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.581+0000 7f79d4f8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.581+0000 7f79d4f8a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0106070 con 0x7f79d0105680 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79ce59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79ce59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60718/0 (socket says 172.21.15.98:60718) 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79ce59c700 1 -- 172.21.15.98:0/1797408437 learned_addr learned my addr 172.21.15.98:0/1797408437 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79ce59c700 1 -- 172.21.15.98:0/1797408437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d0106880 con 0x7f79d0105680 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79ce59c700 1 --2- 172.21.15.98:0/1797408437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f79c40098a0 tx=0x7f79c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4123897d03a1fe8c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79cd59a700 1 -- 172.21.15.98:0/1797408437 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4004160 con 0x7f79d0105680 2024-04-26T04:17:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79cd59a700 1 -- 172.21.15.98:0/1797408437 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f79c4004300 con 0x7f79d0105680 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.582+0000 7f79cd59a700 1 -- 172.21.15.98:0/1797408437 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c40119b0 con 0x7f79d0105680 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 -- 172.21.15.98:0/1797408437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 msgr2=0x7f79d0105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/1797408437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f79c40098a0 tx=0x7f79c4009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 -- 172.21.15.98:0/1797408437 shutdown_connections 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/1797408437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d0105aa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 -- 172.21.15.98:0/1797408437 >> 172.21.15.98:0/1797408437 conn(0x7f79d0100c60 msgr2=0x7f79d0103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 -- 172.21.15.98:0/1797408437 shutdown_connections 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.583+0000 7f79d4f8a700 1 -- 172.21.15.98:0/1797408437 wait complete. 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79d4f8a700 1 Processor -- start 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79d4f8a700 1 -- start start 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79d4f8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79d4f8a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79d0109ab0 con 0x7f79d0105680 2024-04-26T04:17:24.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79ce59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79ce59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60726/0 (socket says 172.21.15.98:60726) 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79ce59c700 1 -- 172.21.15.98:0/3386542152 learned_addr learned my addr 172.21.15.98:0/3386542152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79ce59c700 1 -- 172.21.15.98:0/3386542152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d019ac90 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79ce59c700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f79c4011670 tx=0x7f79c4003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c40038b0 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d019af20 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.584+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f79c40248a0 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c401a670 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d019ba40 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f79c4024430 con 0x7f79d0105680 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 20 mgrc handle_mgr_map mgrmap(e 4) 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 4 mgrc handle_mgr_map Got map version 4 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:24.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79bf7fe700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 0x7f79b803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.585+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d0063850 con 0x7f79d0105680 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.586+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f79c404d3b0 con 0x7f79d0105680 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.586+0000 7f79cdd9b700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 0x7f79b803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.586+0000 7f79cdd9b700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 0x7f79b803b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006fd0 tx=0x7f79c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.591+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f79c40172b0 con 0x7f79d0105680 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.868+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f79d019bcf0 con 0x7f79d0105680 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.868+0000 7f79bf7fe700 1 -- 172.21.15.98:0/3386542152 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f79c400fdd0 con 0x7f79d0105680 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 10 mgrc shutdown 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 msgr2=0x7f79b803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 0x7f79b803b470 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006fd0 tx=0x7f79c0006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 msgr2=0x7f79d019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f79c4011670 tx=0x7f79c4003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.870+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 shutdown_connections 2024-04-26T04:17:24.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.871+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f79b8038fb0 0x7f79b803b470 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:24.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.871+0000 7f79d4f8a700 1 --2- 172.21.15.98:0/3386542152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79d0105680 0x7f79d019a750 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:24.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.871+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 >> 172.21.15.98:0/3386542152 conn(0x7f79d0100c60 msgr2=0x7f79d01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:24.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.871+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 shutdown_connections 2024-04-26T04:17:24.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:17.871+0000 7f79d4f8a700 1 -- 172.21.15.98:0/3386542152 wait complete. 2024-04-26T04:17:24.593 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-04-26T04:17:25.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:25 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/810476268' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T04:17:25.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff448053700 1 Processor -- start 2024-04-26T04:17:25.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff448053700 1 -- start start 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff448053700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff448053700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff4400756b0 con 0x7ff4401071f0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff445def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff445def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45984/0 (socket says 172.21.15.98:45984) 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff445def700 1 -- 172.21.15.98:0/650608975 learned_addr learned my addr 172.21.15.98:0/650608975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff445def700 1 -- 172.21.15.98:0/650608975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff440109b20 con 0x7ff4401071f0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.077+0000 7ff445def700 1 --2- 172.21.15.98:0/650608975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff4300098a0 tx=0x7ff430009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f2cb2c1628a42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff444ded700 1 -- 172.21.15.98:0/650608975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff430004160 con 0x7ff4401071f0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff444ded700 1 -- 172.21.15.98:0/650608975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7ff430004300 con 0x7ff4401071f0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff444ded700 1 -- 172.21.15.98:0/650608975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4300119b0 con 0x7ff4401071f0 2024-04-26T04:17:25.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 -- 172.21.15.98:0/650608975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 msgr2=0x7ff4401095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 --2- 172.21.15.98:0/650608975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff4300098a0 tx=0x7ff430009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 -- 172.21.15.98:0/650608975 shutdown_connections 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 --2- 172.21.15.98:0/650608975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff4401095e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 -- 172.21.15.98:0/650608975 >> 172.21.15.98:0/650608975 conn(0x7ff440100c60 msgr2=0x7ff440103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 -- 172.21.15.98:0/650608975 shutdown_connections 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.078+0000 7ff448053700 1 -- 172.21.15.98:0/650608975 wait complete. 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff448053700 1 Processor -- start 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff448053700 1 -- start start 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff448053700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff448053700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff44010aa90 con 0x7ff4401071f0 2024-04-26T04:17:25.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff445def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff445def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45998/0 (socket says 172.21.15.98:45998) 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff445def700 1 -- 172.21.15.98:0/810476268 learned_addr learned my addr 172.21.15.98:0/810476268 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.079+0000 7ff445def700 1 -- 172.21.15.98:0/810476268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44019ac50 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff445def700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff430009870 tx=0x7ff430011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff430011a40 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7ff4300038b0 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff430019550 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44019aee0 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.080+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44019ba00 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7ff4300197c0 con 0x7ff4401071f0 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 20 mgrc handle_mgr_map mgrmap(e 4) 2024-04-26T04:17:25.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 4 mgrc handle_mgr_map Got map version 4 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 0x7ff42c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff440063850 con 0x7ff4401071f0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff43004d6b0 con 0x7ff4401071f0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff4455ee700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 0x7ff42c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.081+0000 7ff4455ee700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 0x7ff42c03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff43c006fd0 tx=0x7ff43c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.086+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff43004d0d0 con 0x7ff4401071f0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.278+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff44019bd20 con 0x7ff4401071f0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.279+0000 7ff436ffd700 1 -- 172.21.15.98:0/810476268 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7ff43001aaf0 con 0x7ff4401071f0 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 10 mgrc shutdown 2024-04-26T04:17:25.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 msgr2=0x7ff42c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 0x7ff42c03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff43c006fd0 tx=0x7ff43c006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 msgr2=0x7ff44019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff430009870 tx=0x7ff430011820 comp rx=0 tx=0).stop 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 shutdown_connections 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7ff42c039050 0x7ff42c03b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 --2- 172.21.15.98:0/810476268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4401071f0 0x7ff44019a710 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 >> 172.21.15.98:0/810476268 conn(0x7ff440100c60 msgr2=0x7ff440101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 shutdown_connections 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:25.281+0000 7ff448053700 1 -- 172.21.15.98:0/810476268 wait complete. 2024-04-26T04:17:25.880 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-04-26T04:17:26.911 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:26 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/257447863' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T04:17:26.912 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:26 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: ignoring --setuser ceph since I am not root 2024-04-26T04:17:26.912 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:26 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: ignoring --setgroup ceph since I am not root 2024-04-26T04:17:26.912 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:26 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:26.769+0000 7f1f5ff81200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T04:17:27.170 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:26 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:26.954+0000 7f1f5ff81200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:17:27.170 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:27 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:27.055+0000 7f1f5ff81200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T04:17:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9072aa4700 1 Processor -- start 2024-04-26T04:17:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9072aa4700 1 -- start start 2024-04-26T04:17:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9072aa4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:27.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9072aa4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f906c1057d0 con 0x7f906c104e70 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9070840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9070840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46010/0 (socket says 172.21.15.98:46010) 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9070840700 1 -- 172.21.15.98:0/1613341989 learned_addr learned my addr 172.21.15.98:0/1613341989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9070840700 1 -- 172.21.15.98:0/1613341989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c106060 con 0x7f906c104e70 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.286+0000 7f9070840700 1 --2- 172.21.15.98:0/1613341989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90600098a0 tx=0x7f9060009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d00dc505308471 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f906b7fe700 1 -- 172.21.15.98:0/1613341989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9060004160 con 0x7f906c104e70 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f906b7fe700 1 -- 172.21.15.98:0/1613341989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9060004300 con 0x7f906c104e70 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f906b7fe700 1 -- 172.21.15.98:0/1613341989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90600119b0 con 0x7f906c104e70 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 -- 172.21.15.98:0/1613341989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 msgr2=0x7f906c105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 --2- 172.21.15.98:0/1613341989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90600098a0 tx=0x7f9060009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 -- 172.21.15.98:0/1613341989 shutdown_connections 2024-04-26T04:17:27.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 --2- 172.21.15.98:0/1613341989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c105290 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 -- 172.21.15.98:0/1613341989 >> 172.21.15.98:0/1613341989 conn(0x7f906c1004a0 msgr2=0x7f906c1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 -- 172.21.15.98:0/1613341989 shutdown_connections 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.287+0000 7f9072aa4700 1 -- 172.21.15.98:0/1613341989 wait complete. 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9072aa4700 1 Processor -- start 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9072aa4700 1 -- start start 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9072aa4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9072aa4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f906c105950 con 0x7f906c104e70 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9070840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9070840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46026/0 (socket says 172.21.15.98:46026) 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9070840700 1 -- 172.21.15.98:0/257447863 learned_addr learned my addr 172.21.15.98:0/257447863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:27.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9070840700 1 -- 172.21.15.98:0/257447863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c19ace0 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.288+0000 7f9070840700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9060011670 tx=0x7f9060011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.289+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90600036a0 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.289+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9060003840 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.289+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906c19af70 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.289+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9060019580 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.289+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906c19dc60 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f9060019a20 con 0x7f906c104e70 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 20 mgrc handle_mgr_map mgrmap(e 4) 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 4 mgrc handle_mgr_map Got map version 4 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:27.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 0x7f905c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906c063850 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f906004d7e0 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.290+0000 7f906bfff700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 0x7f905c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.291+0000 7f906bfff700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 0x7f905c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9058006fd0 tx=0x7f9058006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.296+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f906004a840 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.551+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f906c02daf0 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.603+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f9060033020 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.604+0000 7f9069ffb700 1 -- 172.21.15.98:0/257447863 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f906002d960 con 0x7f906c104e70 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.604+0000 7f9069ffb700 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.604+0000 7f9069ffb700 4 mgrc handle_mgr_map Got map version 5 2024-04-26T04:17:27.184 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.604+0000 7f9069ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 10 mgrc shutdown 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 msgr2=0x7f905c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 0x7f905c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9058006fd0 tx=0x7f9058006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 msgr2=0x7f906c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9060011670 tx=0x7f9060011820 comp rx=0 tx=0).stop 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 shutdown_connections 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f905c039000 0x7f905c03b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 --2- 172.21.15.98:0/257447863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f906c104e70 0x7f906c19a7a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 >> 172.21.15.98:0/257447863 conn(0x7f906c1004a0 msgr2=0x7f906c102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 shutdown_connections 2024-04-26T04:17:27.185 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:26.605+0000 7f9072aa4700 1 -- 172.21.15.98:0/257447863 wait complete. 2024-04-26T04:17:27.439 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:27 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:27.370+0000 7f1f5ff81200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T04:17:27.791 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:27 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:27.456+0000 7f1f5ff81200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T04:17:27.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:27 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/257447863' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T04:17:27.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:27 smithi098 ceph-mon[27184]: mgrmap e5: a(active, since 12s) 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "a", 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:28.486 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf3c9a700 1 Processor -- start 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf3c9a700 1 -- start start 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf3c9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf3c9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec079e40 con 0x7f6cec107200 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf1a36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf1a36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46052/0 (socket says 172.21.15.98:46052) 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.658+0000 7f6cf1a36700 1 -- 172.21.15.98:0/288976980 learned_addr learned my addr 172.21.15.98:0/288976980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.659+0000 7f6cf1a36700 1 -- 172.21.15.98:0/288976980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec109b30 con 0x7f6cec107200 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.659+0000 7f6cf1a36700 1 --2- 172.21.15.98:0/288976980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80098a0 tx=0x7f6ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95804331597825b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.659+0000 7f6cf0a34700 1 -- 172.21.15.98:0/288976980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce8004160 con 0x7f6cec107200 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.659+0000 7f6cf0a34700 1 -- 172.21.15.98:0/288976980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f6ce8004300 con 0x7f6cec107200 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.659+0000 7f6cf0a34700 1 -- 172.21.15.98:0/288976980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce80119b0 con 0x7f6cec107200 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/288976980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 msgr2=0x7f6cec1095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/288976980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80098a0 tx=0x7f6ce8009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/288976980 shutdown_connections 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/288976980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec1095f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/288976980 >> 172.21.15.98:0/288976980 conn(0x7f6cec100c50 msgr2=0x7f6cec103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/288976980 shutdown_connections 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/288976980 wait complete. 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 Processor -- start 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.660+0000 7f6cf3c9a700 1 -- start start 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf3c9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf3c9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec10aa90 con 0x7f6cec107200 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf1a36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf1a36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46054/0 (socket says 172.21.15.98:46054) 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf1a36700 1 -- 172.21.15.98:0/1124840028 learned_addr learned my addr 172.21.15.98:0/1124840028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf1a36700 1 -- 172.21.15.98:0/1124840028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec106fc0 con 0x7f6cec107200 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf1a36700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8011670 tx=0x7f6ce8011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce8011a70 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f6ce80038b0 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ce8019560 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cec1051b0 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.661+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cec1056f0 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.662+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cec063850 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.662+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f6ce8019700 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.662+0000 7f6ce2ffd700 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6ce2ffd700 4 mgrc handle_mgr_map Got map version 5 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6ce2ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6ce2ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6ce2ffd700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 0x7f6cd803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6ce804d510 con 0x7f6cec107200 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6cf1235700 1 -- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 msgr2=0x7f6cd803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:28.489 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.663+0000 7f6cf1235700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 0x7f6cd803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.667+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6ce804cf30 con 0x7f6cec107200 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.864+0000 7f6cf1235700 1 -- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 msgr2=0x7f6cd803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.864+0000 7f6cf1235700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 0x7f6cd803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.960+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f6cec105ce0 con 0x7f6cec107200 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.961+0000 7f6ce2ffd700 1 -- 172.21.15.98:0/1124840028 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f6ce8029d50 con 0x7f6cec107200 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 10 mgrc shutdown 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 msgr2=0x7f6cd803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 0x7f6cd803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 msgr2=0x7f6cec106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8011670 tx=0x7f6ce8011820 comp rx=0 tx=0).stop 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 shutdown_connections 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f6cd8039050 0x7f6cd803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 --2- 172.21.15.98:0/1124840028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cec107200 0x7f6cec106a80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:28.490 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 >> 172.21.15.98:0/1124840028 conn(0x7f6cec100c50 msgr2=0x7f6cec101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:28.491 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 shutdown_connections 2024-04-26T04:17:28.491 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:27.963+0000 7f6cf3c9a700 1 -- 172.21.15.98:0/1124840028 wait complete. 2024-04-26T04:17:28.491 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-26T04:17:28.491 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-04-26T04:17:28.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:28 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1124840028' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T04:17:28.897 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:28 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:28.655+0000 7f1f5ff81200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T04:17:28.897 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:28 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:28.828+0000 7f1f5ff81200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:29.148 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:29 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:29.004+0000 7f1f5ff81200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T04:17:29.541 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:29 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:29.288+0000 7f1f5ff81200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T04:17:29.541 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:29 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:29.380+0000 7f1f5ff81200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T04:17:30.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:29 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:29.579+0000 7f1f5ff81200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T04:17:30.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:29 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:29.774+0000 7f1f5ff81200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:17:30.541 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:30 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:30.382+0000 7f1f5ff81200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T04:17:31.092 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:30 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:30.799+0000 7f1f5ff81200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T04:17:31.093 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:30 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:30.894+0000 7f1f5ff81200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T04:17:31.093 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:31.091+0000 7f1f5ff81200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T04:17:31.938 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:31.609+0000 7f1f5ff81200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T04:17:31.939 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:31.699+0000 7f1f5ff81200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T04:17:32.279 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:31.937+0000 7f1f5ff81200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T04:17:32.279 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:32.070+0000 7f1f5ff81200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T04:17:32.279 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:32.163+0000 7f1f5ff81200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T04:17:32.279 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:32.278+0000 7f1f5ff81200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:32.541 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:32.369+0000 7f1f5ff81200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T04:17:33.442 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:33.174+0000 7f1f5ff81200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T04:17:33.443 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:33.389+0000 7f1f5ff81200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T04:17:33.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: Active manager daemon a restarted 2024-04-26T04:17:33.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: Activating manager daemon a 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: mgrmap e6: a(active, starting, since 0.00333969s) 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "mgr metadata", "who": "a", "id": "a"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: Manager daemon a is now available 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/mirror_snapshot_schedule"}]: dispatch 2024-04-26T04:17:33.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:33 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/trash_purge_schedule"}]: dispatch 2024-04-26T04:17:34.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:34 smithi098 ceph-mon[27184]: Found migration_current of "None". Setting to last migration. 2024-04-26T04:17:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:34 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:34 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:34 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:17:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:34 smithi098 ceph-mon[27184]: mgrmap e7: a(active, since 1.00565s) 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f1541f04700 1 Processor -- start 2024-04-26T04:17:34.963 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f1541f04700 1 -- start start 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f1541f04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f1541f04700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f153c106080 con 0x7f153c105690 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f153b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f153b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44598/0 (socket says 172.21.15.98:44598) 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f153b7fe700 1 -- 172.21.15.98:0/1152268240 learned_addr learned my addr 172.21.15.98:0/1152268240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.942+0000 7f153b7fe700 1 -- 172.21.15.98:0/1152268240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153c106890 con 0x7f153c105690 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f153b7fe700 1 --2- 172.21.15.98:0/1152268240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f152c0098a0 tx=0x7f152c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2d3c6068faa017 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f153a7fc700 1 -- 172.21.15.98:0/1152268240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f152c004160 con 0x7f153c105690 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f153a7fc700 1 -- 172.21.15.98:0/1152268240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f152c004300 con 0x7f153c105690 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f153a7fc700 1 -- 172.21.15.98:0/1152268240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f152c0119b0 con 0x7f153c105690 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f1541f04700 1 -- 172.21.15.98:0/1152268240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 msgr2=0x7f153c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f1541f04700 1 --2- 172.21.15.98:0/1152268240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f152c0098a0 tx=0x7f152c009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f1541f04700 1 -- 172.21.15.98:0/1152268240 shutdown_connections 2024-04-26T04:17:34.964 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f1541f04700 1 --2- 172.21.15.98:0/1152268240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c105ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.943+0000 7f1541f04700 1 -- 172.21.15.98:0/1152268240 >> 172.21.15.98:0/1152268240 conn(0x7f153c100c70 msgr2=0x7f153c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 -- 172.21.15.98:0/1152268240 shutdown_connections 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 -- 172.21.15.98:0/1152268240 wait complete. 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 Processor -- start 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 -- start start 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.944+0000 7f1541f04700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f153c109ac0 con 0x7f153c105690 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f153b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f153b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44612/0 (socket says 172.21.15.98:44612) 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f153b7fe700 1 -- 172.21.15.98:0/443824613 learned_addr learned my addr 172.21.15.98:0/443824613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f153b7fe700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153c19d300 con 0x7f153c105690 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f153b7fe700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f152c011670 tx=0x7f152c011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f152c0038b0 con 0x7f153c105690 2024-04-26T04:17:34.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f152c0248a0 con 0x7f153c105690 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153c19d4d0 con 0x7f153c105690 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f152c01a6a0 con 0x7f153c105690 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.945+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f153c19d8f0 con 0x7f153c105690 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1541f04700 20 mgrc start_tell_command target: cmd: [{"prefix": "get_command_descriptions"}] 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1541f04700 5 mgrc start_tell_command no mgr session (no running mgr daemon?), or not active mgr, waiting 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f152c01a840 con 0x7f153c105690 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 20 mgrc handle_mgr_map mgrmap(e 5) 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 4 mgrc handle_mgr_map Got map version 5 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 10 mgrc reconnect resending 0 (tell) 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f152403bc70 con 0x7f15240390a0 2024-04-26T04:17:34.966 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f152c04d2b0 con 0x7f153c105690 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f153affd700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:28.946+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:29.147+0000 7f153affd700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:29.147+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:29.548+0000 7f153affd700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:29.548+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:30.349+0000 7f153affd700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:30.349+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:31.950+0000 7f153affd700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:31.950+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f152c024a50 con 0x7f153c105690 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 20 mgrc handle_mgr_map mgrmap(e 6) 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 4 mgrc handle_mgr_map Got map version 6 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:17:34.967 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 4 mgrc reconnect Terminating session with v2:172.21.15.98:6800/2639234633 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 msgr2=0x7f152403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:33.392+0000 7f1538ff9700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.393+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f152c04ed40 con 0x7f153c105690 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.393+0000 7f1538ff9700 20 mgrc handle_mgr_map mgrmap(e 7) 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 4 mgrc handle_mgr_map Got map version 7 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 0x7f152403edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 10 mgrc reconnect resending 0 (tell) 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f152c009530 con 0x7f152403c9d0 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.394+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 0x7f152403edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.395+0000 7f153affd700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 0x7f152403edc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1530003a10 tx=0x7f15300092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.395+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f152c009530 con 0x7f152403c9d0 2024-04-26T04:17:34.968 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.395+0000 7f1538ff9700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 20 mgrc start_tell_command target: cmd: [{"prefix": "mgr_status"}] 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f153c063850 con 0x7f152403c9d0 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1538ff9700 1 -- 172.21.15.98:0/443824613 <== mgr.14118 v2:172.21.15.98:6800/142088859 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f153c063850 con 0x7f152403c9d0 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1538ff9700 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 10 mgrc shutdown 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 msgr2=0x7f152403edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 0x7f152403edc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1530003a10 tx=0x7f15300092b0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 msgr2=0x7f153c079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f152c011670 tx=0x7f152c011820 comp rx=0 tx=0).stop 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 shutdown_connections 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f153c105690 0x7f153c079a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f152403c9d0 0x7f152403edc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.969 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 --2- 172.21.15.98:0/443824613 >> [v2:172.21.15.98:6800/2639234633,v1:172.21.15.98:6801/2639234633] conn(0x7f15240390a0 0x7f152403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:34.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.402+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 >> 172.21.15.98:0/443824613 conn(0x7f153c100c70 msgr2=0x7f153c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:34.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.403+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 shutdown_connections 2024-04-26T04:17:34.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:34.403+0000 7f1541f04700 1 -- 172.21.15.98:0/443824613 wait complete. 2024-04-26T04:17:34.970 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-04-26T04:17:34.970 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: [26/Apr/2024:04:17:34] ENGINE Bus STARTING 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: [26/Apr/2024:04:17:34] ENGINE Serving on http://172.21.15.98:8765 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: [26/Apr/2024:04:17:34] ENGINE Serving on https://172.21.15.98:7150 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: [26/Apr/2024:04:17:34] ENGINE Bus STARTED 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T04:17:35.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:35 smithi098 ceph-mon[27184]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T04:17:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.354+0000 7efede22b700 1 Processor -- start 2024-04-26T04:17:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.354+0000 7efede22b700 1 -- start start 2024-04-26T04:17:36.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.354+0000 7efede22b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.354+0000 7efede22b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efed8105520 con 0x7efed8104b30 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.354+0000 7efed77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44678/0 (socket says 172.21.15.98:44678) 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed77fe700 1 -- 172.21.15.98:0/801568594 learned_addr learned my addr 172.21.15.98:0/801568594 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed77fe700 1 -- 172.21.15.98:0/801568594 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed8107ac0 con 0x7efed8104b30 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed77fe700 1 --2- 172.21.15.98:0/801568594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec80098a0 tx=0x7efec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e70278790b67db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed6ffd700 1 -- 172.21.15.98:0/801568594 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efec8004160 con 0x7efed8104b30 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed6ffd700 1 -- 172.21.15.98:0/801568594 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7efec8004300 con 0x7efed8104b30 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.355+0000 7efed6ffd700 1 -- 172.21.15.98:0/801568594 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efec80119b0 con 0x7efed8104b30 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- 172.21.15.98:0/801568594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 msgr2=0x7efed8104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 --2- 172.21.15.98:0/801568594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efec80098a0 tx=0x7efec8009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- 172.21.15.98:0/801568594 shutdown_connections 2024-04-26T04:17:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 --2- 172.21.15.98:0/801568594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed8104f50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- 172.21.15.98:0/801568594 >> 172.21.15.98:0/801568594 conn(0x7efed81004a0 msgr2=0x7efed81028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- 172.21.15.98:0/801568594 shutdown_connections 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- 172.21.15.98:0/801568594 wait complete. 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 Processor -- start 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.356+0000 7efede22b700 1 -- start start 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efede22b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efede22b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efed810acf0 con 0x7efed8104b30 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44686/0 (socket says 172.21.15.98:44686) 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed77fe700 1 -- 172.21.15.98:0/3823006260 learned_addr learned my addr 172.21.15.98:0/3823006260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed77fe700 1 -- 172.21.15.98:0/3823006260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed819ac40 con 0x7efed8104b30 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed77fe700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efec8011670 tx=0x7efec8011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efec80038b0 con 0x7efed8104b30 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7efec80248a0 con 0x7efed8104b30 2024-04-26T04:17:36.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed819aed0 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efec801a6a0 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.357+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed819dbc0 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7efec801a840 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 20 mgrc handle_mgr_map mgrmap(e 7) 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed810ae70 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 4 mgrc handle_mgr_map Got map version 7 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 0x7efec403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efec804d1c0 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.358+0000 7efeceffd700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 0x7efec403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.359+0000 7efeceffd700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 0x7efec403b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.363+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efec8019c70 con 0x7efed8104b30 2024-04-26T04:17:36.152 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.449+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7efec8024a50 con 0x7efed8104b30 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.449+0000 7efed57fa700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.449+0000 7efed57fa700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.449+0000 7efed57fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.582+0000 7efede22b700 20 mgrc start_command cmd: [{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}] 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.582+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7efed8063850 con 0x7efec4038f80 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.586+0000 7efed57fa700 1 -- 172.21.15.98:0/3823006260 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efed8063850 con 0x7efec4038f80 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.586+0000 7efed57fa700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 10 mgrc shutdown 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 msgr2=0x7efec403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 0x7efec403b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 msgr2=0x7efed819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efec8011670 tx=0x7efec8011820 comp rx=0 tx=0).stop 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 shutdown_connections 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7efec4038f80 0x7efec403b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:36.153 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 --2- 172.21.15.98:0/3823006260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efed8104b30 0x7efed819a700 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 >> 172.21.15.98:0/3823006260 conn(0x7efed81004a0 msgr2=0x7efed8100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 shutdown_connections 2024-04-26T04:17:36.154 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:35.589+0000 7efede22b700 1 -- 172.21.15.98:0/3823006260 wait complete. 2024-04-26T04:17:36.759 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:36 smithi098 ceph-mon[27184]: mgrmap e8: a(active, since 2s) 2024-04-26T04:17:36.759 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:36 smithi098 ceph-mon[27184]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:36.759 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:36 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:36.759 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:36 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:17:37.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T04:17:37.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.549+0000 7f5512add700 1 Processor -- start 2024-04-26T04:17:37.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.549+0000 7f5512add700 1 -- start start 2024-04-26T04:17:37.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.549+0000 7f5512add700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:37.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.549+0000 7f5512add700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f550c07a8e0 con 0x7f550c07beb0 2024-04-26T04:17:37.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.549+0000 7f5510879700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:37.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f5510879700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44698/0 (socket says 172.21.15.98:44698) 2024-04-26T04:17:37.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f5510879700 1 -- 172.21.15.98:0/3259612862 learned_addr learned my addr 172.21.15.98:0/3259612862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:37.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f5510879700 1 -- 172.21.15.98:0/3259612862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c07aa60 con 0x7f550c07beb0 2024-04-26T04:17:37.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f5510879700 1 --2- 172.21.15.98:0/3259612862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55000098a0 tx=0x7f5500009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3239924dfa82eb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f550b7fe700 1 -- 172.21.15.98:0/3259612862 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5500004160 con 0x7f550c07beb0 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f550b7fe700 1 -- 172.21.15.98:0/3259612862 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f5500004300 con 0x7f550c07beb0 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.550+0000 7f550b7fe700 1 -- 172.21.15.98:0/3259612862 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55000119b0 con 0x7f550c07beb0 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- 172.21.15.98:0/3259612862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 msgr2=0x7f550c07a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 --2- 172.21.15.98:0/3259612862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55000098a0 tx=0x7f5500009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:37.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- 172.21.15.98:0/3259612862 shutdown_connections 2024-04-26T04:17:37.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 --2- 172.21.15.98:0/3259612862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c07a310 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:37.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- 172.21.15.98:0/3259612862 >> 172.21.15.98:0/3259612862 conn(0x7f550c101d30 msgr2=0x7f550c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:37.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- 172.21.15.98:0/3259612862 shutdown_connections 2024-04-26T04:17:37.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- 172.21.15.98:0/3259612862 wait complete. 2024-04-26T04:17:37.258 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 Processor -- start 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 -- start start 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.551+0000 7f5512add700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5512add700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f550c1088c0 con 0x7f550c07beb0 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5510879700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5510879700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44714/0 (socket says 172.21.15.98:44714) 2024-04-26T04:17:37.259 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5510879700 1 -- 172.21.15.98:0/2719854954 learned_addr learned my addr 172.21.15.98:0/2719854954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5510879700 1 -- 172.21.15.98:0/2719854954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550c1a1290 con 0x7f550c07beb0 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5510879700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5500011670 tx=0x7f5500003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55000038b0 con 0x7f550c07beb0 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f55000248a0 con 0x7f550c07beb0 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550001a670 con 0x7f550c07beb0 2024-04-26T04:17:37.260 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550c1a1520 con 0x7f550c07beb0 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.552+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550c1a41b0 con 0x7f550c07beb0 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5500024a50 con 0x7f550c07beb0 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:37.261 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:37.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550c063850 con 0x7f550c07beb0 2024-04-26T04:17:37.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 0x7f54fc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:37.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.553+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f550004d350 con 0x7f550c07beb0 2024-04-26T04:17:37.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.554+0000 7f550bfff700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 0x7f54fc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:37.262 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.554+0000 7f550bfff700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 0x7f54fc03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f54f8006fd0 tx=0x7f54f8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.559+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f550002d3f0 con 0x7f550c07beb0 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.758+0000 7f5512add700 20 mgrc start_command cmd: [{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}] 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.758+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f550c1a1a00 con 0x7f54fc039050 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.759+0000 7f5509ffb700 1 -- 172.21.15.98:0/2719854954 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f550c1a1a00 con 0x7f54fc039050 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.759+0000 7f5509ffb700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:37.263 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 10 mgrc shutdown 2024-04-26T04:17:37.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 msgr2=0x7f54fc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:37.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 0x7f54fc03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f54f8006fd0 tx=0x7f54f8006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:37.265 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 msgr2=0x7f550c1a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5500011670 tx=0x7f5500003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 shutdown_connections 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f54fc039050 0x7f54fc03b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 --2- 172.21.15.98:0/2719854954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f550c07beb0 0x7f550c1a0d50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 >> 172.21.15.98:0/2719854954 conn(0x7f550c101d30 msgr2=0x7f550c197a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:37.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 shutdown_connections 2024-04-26T04:17:37.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:36.761+0000 7f5512add700 1 -- 172.21.15.98:0/2719854954 wait complete. 2024-04-26T04:17:37.267 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-04-26T04:17:37.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:37 smithi098 ceph-mon[27184]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:38.085 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: Generating public/private rsa key pair. 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: Your identification has been saved in /tmp/tmpv8_u1m9a/key. 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: Your public key has been saved in /tmp/tmpv8_u1m9a/key.pub. 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: The key fingerprint is: 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: SHA256:ve4/0CXEWS926jPDKmLyW7a6uQ9pJ9fAt9/ydn09dbU ceph-b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: The key's randomart image is: 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: +---[RSA 3072]----+ 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | . o. | 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | + . | 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | . o o| 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | o ...+.| 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | S +..o. o| 2024-04-26T04:17:38.086 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | ..=.+ Eo| 2024-04-26T04:17:38.087 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | = *.o * =| 2024-04-26T04:17:38.087 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | ..oX...o.**| 2024-04-26T04:17:38.087 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: | +BO=oo..+*| 2024-04-26T04:17:38.087 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:38 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: +----[SHA256]-----+ 2024-04-26T04:17:38.623 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5b484d700 1 Processor -- start 2024-04-26T04:17:38.623 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5b484d700 1 -- start start 2024-04-26T04:17:38.623 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5b484d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5b484d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b01071e0 con 0x7fd5b01067f0 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44726/0 (socket says 172.21.15.98:44726) 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5ae59c700 1 -- 172.21.15.98:0/2458252570 learned_addr learned my addr 172.21.15.98:0/2458252570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.622+0000 7fd5ae59c700 1 -- 172.21.15.98:0/2458252570 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b01079f0 con 0x7fd5b01067f0 2024-04-26T04:17:38.624 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5ae59c700 1 --2- 172.21.15.98:0/2458252570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4745407f0a06f16a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5ad59a700 1 -- 172.21.15.98:0/2458252570 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4004160 con 0x7fd5b01067f0 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5ad59a700 1 -- 172.21.15.98:0/2458252570 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7fd5a4004300 con 0x7fd5b01067f0 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5ad59a700 1 -- 172.21.15.98:0/2458252570 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a40119b0 con 0x7fd5b01067f0 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 -- 172.21.15.98:0/2458252570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 msgr2=0x7fd5b0106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 --2- 172.21.15.98:0/2458252570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098a0 tx=0x7fd5a4009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:38.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 -- 172.21.15.98:0/2458252570 shutdown_connections 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 --2- 172.21.15.98:0/2458252570 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b0106c10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 -- 172.21.15.98:0/2458252570 >> 172.21.15.98:0/2458252570 conn(0x7fd5b0101df0 msgr2=0x7fd5b0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 -- 172.21.15.98:0/2458252570 shutdown_connections 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.623+0000 7fd5b484d700 1 -- 172.21.15.98:0/2458252570 wait complete. 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5b484d700 1 Processor -- start 2024-04-26T04:17:38.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5b484d700 1 -- start start 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5b484d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5b484d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b010ac20 con 0x7fd5b01067f0 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44734/0 (socket says 172.21.15.98:44734) 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.624+0000 7fd5ae59c700 1 -- 172.21.15.98:0/1413015333 learned_addr learned my addr 172.21.15.98:0/1413015333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd5ae59c700 1 -- 172.21.15.98:0/1413015333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b019ab50 con 0x7fd5b01067f0 2024-04-26T04:17:38.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd5ae59c700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4011670 tx=0x7fd5a4011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a40036a0 con 0x7fd5b01067f0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b019ade0 con 0x7fd5b01067f0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7fd5a4003840 con 0x7fd5b01067f0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a4019580 con 0x7fd5b01067f0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.625+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b019dad0 con 0x7fd5b01067f0 2024-04-26T04:17:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd5a40198c0 con 0x7fd5b01067f0 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b010ada0 con 0x7fd5b01067f0 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:38.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 0x7fd59803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd5a404d770 con 0x7fd5b01067f0 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.626+0000 7fd5add9b700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 0x7fd59803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.627+0000 7fd5add9b700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 0x7fd59803b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0006fd0 tx=0x7fd5a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.631+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd5a404d0c0 con 0x7fd5b01067f0 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.829+0000 7fd5b484d700 20 mgrc start_command cmd: [{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}] 2024-04-26T04:17:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:37.829+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd5b0102ff0 con 0x7fd598039000 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.082+0000 7fd59f7fe700 1 -- 172.21.15.98:0/1413015333 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd5b0102ff0 con 0x7fd598039000 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.082+0000 7fd59f7fe700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 10 mgrc shutdown 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 msgr2=0x7fd59803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 0x7fd59803b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0006fd0 tx=0x7fd5a0006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:38.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 msgr2=0x7fd5b019a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4011670 tx=0x7fd5a4011820 comp rx=0 tx=0).stop 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 shutdown_connections 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fd598039000 0x7fd59803b4c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 --2- 172.21.15.98:0/1413015333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5b01067f0 0x7fd5b019a610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 >> 172.21.15.98:0/1413015333 conn(0x7fd5b0101df0 msgr2=0x7fd5b01028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 shutdown_connections 2024-04-26T04:17:38.632 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:38.084+0000 7fd5b484d700 1 -- 172.21.15.98:0/1413015333 wait complete. 2024-04-26T04:17:39.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:39 smithi098 ceph-mon[27184]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:39.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:39 smithi098 ceph-mon[27184]: Generating ssh key... 2024-04-26T04:17:39.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:39 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:39.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:39 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1tKQV6ocTg3ya8EUdsmK+Yf+rBmfazvvZ897Eylrm5BZJW/4aZb32uUwJ2fOriRl7NmJ9NKV9Juh4Co3xudlym7phhnmgaatkZouI08KB3q7CtG0tNttnTeLrTRle47wqXD9PRDvhEO3Fgqj5UmsZ674B4ThOqIl6X7Je1hsFvcX7hcpwWNtIZaikXQC6YkeRvugfASk2YZbl/be8XJCG9mHwUXmOteb6TX8iOcagvQ3b7YeBzrML46A58dzCXeWaIt96iAyGPe3QUSV6i2pZOEbt/Y1TSQjhW17CZUgK8IvGWPlPX7GUQIXaJ5S0ib55Y7Hbl4Wc48Bq5Da8ZaT/8NZKCA/kVLqg+6rkavwfteeOfvyUuR0VWktTeB9d3GkIUGVJVJPh7vpIHuZvi9fVK6ljkzTkXi0VMxacO3K74xT5/uckoO902a+CKl4PAIfduUgN9RtJelAbnLXUa4leGVMOi71QpaLvBojeTbuPyV3XewsoEM2X2BnJ2CFiz1E= ceph-b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:17:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.036+0000 7f999f2df700 1 Processor -- start 2024-04-26T04:17:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.036+0000 7f999f2df700 1 -- start start 2024-04-26T04:17:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999f2df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:39.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999f2df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9998106070 con 0x7f9998105680 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999d07b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999d07b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37692/0 (socket says 172.21.15.98:37692) 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999d07b700 1 -- 172.21.15.98:0/3414775065 learned_addr learned my addr 172.21.15.98:0/3414775065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999d07b700 1 -- 172.21.15.98:0/3414775065 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9998106880 con 0x7f9998105680 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f999d07b700 1 --2- 172.21.15.98:0/3414775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99880098a0 tx=0x7f9988009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0973342caf3913b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f998ffff700 1 -- 172.21.15.98:0/3414775065 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9988004160 con 0x7f9998105680 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f998ffff700 1 -- 172.21.15.98:0/3414775065 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9988004300 con 0x7f9998105680 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.037+0000 7f998ffff700 1 -- 172.21.15.98:0/3414775065 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99880119b0 con 0x7f9998105680 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 -- 172.21.15.98:0/3414775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 msgr2=0x7f9998105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 --2- 172.21.15.98:0/3414775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99880098a0 tx=0x7f9988009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 -- 172.21.15.98:0/3414775065 shutdown_connections 2024-04-26T04:17:39.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 --2- 172.21.15.98:0/3414775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f9998105aa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 -- 172.21.15.98:0/3414775065 >> 172.21.15.98:0/3414775065 conn(0x7f9998100c60 msgr2=0x7f9998103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 -- 172.21.15.98:0/3414775065 shutdown_connections 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 -- 172.21.15.98:0/3414775065 wait complete. 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.038+0000 7f999f2df700 1 Processor -- start 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999f2df700 1 -- start start 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999f2df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999f2df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9998109ab0 con 0x7f9998105680 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999d07b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999d07b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37700/0 (socket says 172.21.15.98:37700) 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999d07b700 1 -- 172.21.15.98:0/3606180381 learned_addr learned my addr 172.21.15.98:0/3606180381 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999d07b700 1 -- 172.21.15.98:0/3606180381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999819ad10 con 0x7f9998105680 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999d07b700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9988006af0 tx=0x7f9988003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99880038b0 con 0x7f9998105680 2024-04-26T04:17:39.822 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f998801a8a0 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99880246e0 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999819afa0 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.039+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999819bac0 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9988022020 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 0x7f998403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f998804d190 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9998063850 con 0x7f9998105680 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.040+0000 7f999c87a700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 0x7f998403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.041+0000 7f999c87a700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 0x7f998403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9994006fd0 tx=0x7f9994006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:39.823 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.045+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f998802a2a0 con 0x7f9998105680 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.242+0000 7f999f2df700 20 mgrc start_command cmd: [{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}] 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.242+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f999819bd40 con 0x7f99840390a0 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.243+0000 7f998e7fc700 1 -- 172.21.15.98:0/3606180381 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f999819bd40 con 0x7f99840390a0 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.243+0000 7f998e7fc700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 10 mgrc shutdown 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 msgr2=0x7f998403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 0x7f998403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9994006fd0 tx=0x7f9994006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 msgr2=0x7f999819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9988006af0 tx=0x7f9988003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:39.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 shutdown_connections 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f99840390a0 0x7f998403b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 --2- 172.21.15.98:0/3606180381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998105680 0x7f999819a7d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 >> 172.21.15.98:0/3606180381 conn(0x7f9998100c60 msgr2=0x7f9998101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 shutdown_connections 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:39.245+0000 7f999f2df700 1 -- 172.21.15.98:0/3606180381 wait complete. 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T04:17:39.825 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-04-26T04:17:40.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:40 smithi098 ceph-mon[27184]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:41.267 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:41 smithi098 ceph-mon[27184]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:42.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:42 smithi098 ceph-mon[27184]: Deploying cephadm binary to smithi098 2024-04-26T04:17:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-04-26T04:17:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.213+0000 7f87e5bab700 1 Processor -- start 2024-04-26T04:17:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.213+0000 7f87e5bab700 1 -- start start 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.213+0000 7f87e5bab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.213+0000 7f87e5bab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0106050 con 0x7f87e0105660 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.213+0000 7f87df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37708/0 (socket says 172.21.15.98:37708) 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87df7fe700 1 -- 172.21.15.98:0/3598114306 learned_addr learned my addr 172.21.15.98:0/3598114306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87df7fe700 1 -- 172.21.15.98:0/3598114306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e0106860 con 0x7f87e0105660 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87df7fe700 1 --2- 172.21.15.98:0/3598114306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87d00098a0 tx=0x7f87d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfffa2eff071d3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87deffd700 1 -- 172.21.15.98:0/3598114306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d0004160 con 0x7f87e0105660 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87deffd700 1 -- 172.21.15.98:0/3598114306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f87d0004300 con 0x7f87e0105660 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.214+0000 7f87deffd700 1 -- 172.21.15.98:0/3598114306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d00119b0 con 0x7f87e0105660 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- 172.21.15.98:0/3598114306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 msgr2=0x7f87e0105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 --2- 172.21.15.98:0/3598114306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87d00098a0 tx=0x7f87d0009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- 172.21.15.98:0/3598114306 shutdown_connections 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 --2- 172.21.15.98:0/3598114306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e0105a80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- 172.21.15.98:0/3598114306 >> 172.21.15.98:0/3598114306 conn(0x7f87e0100c20 msgr2=0x7f87e0103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- 172.21.15.98:0/3598114306 shutdown_connections 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- 172.21.15.98:0/3598114306 wait complete. 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 Processor -- start 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.215+0000 7f87e5bab700 1 -- start start 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87e5bab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87e5bab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0109a90 con 0x7f87e0105660 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87df7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37714/0 (socket says 172.21.15.98:37714) 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87df7fe700 1 -- 172.21.15.98:0/220166001 learned_addr learned my addr 172.21.15.98:0/220166001 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:45.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87df7fe700 1 -- 172.21.15.98:0/220166001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e0190c20 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87df7fe700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011670 tx=0x7f87d0011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d00036a0 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e018ee60 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f87d0003840 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87d0019580 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.216+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e018f340 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.217+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f87d0019840 con 0x7f87e0105660 2024-04-26T04:17:45.006 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.217+0000 7f87dd7fa700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.217+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e0063850 con 0x7f87e0105660 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87dd7fa700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87dd7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87dd7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87dd7fa700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 0x7f87cc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f87d004d7a0 con 0x7f87e0105660 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87d6ffd700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 0x7f87cc03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.218+0000 7f87d6ffd700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 0x7f87cc03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87c8006fd0 tx=0x7f87c8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.222+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87d000fda0 con 0x7f87e0105660 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.422+0000 7f87e5bab700 20 mgrc start_command cmd: [{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}] 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:40.422+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7f87e0194350 con 0x7f87cc039000 2024-04-26T04:17:45.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.450+0000 7f87dd7fa700 1 -- 172.21.15.98:0/220166001 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f87e0194350 con 0x7f87cc039000 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.450+0000 7f87dd7fa700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.452+0000 7f87e5bab700 10 mgrc shutdown 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.452+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 msgr2=0x7f87cc03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.452+0000 7f87e5bab700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 0x7f87cc03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87c8006fd0 tx=0x7f87c8006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.452+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 msgr2=0x7f87e01906e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.452+0000 7f87e5bab700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f87d0011670 tx=0x7f87d0011820 comp rx=0 tx=0).stop 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 shutdown_connections 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f87cc039000 0x7f87cc03b4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 --2- 172.21.15.98:0/220166001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87e0105660 0x7f87e01906e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 >> 172.21.15.98:0/220166001 conn(0x7f87e0100c20 msgr2=0x7f87e0101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 shutdown_connections 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:44.453+0000 7f87e5bab700 1 -- 172.21.15.98:0/220166001 wait complete. 2024-04-26T04:17:45.008 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-04-26T04:17:45.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:45 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:45.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:45 smithi098 ceph-mon[27184]: Added host smithi098 2024-04-26T04:17:45.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:45 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:17:46.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.450+0000 7f932c5ba700 1 Processor -- start 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.450+0000 7f932c5ba700 1 -- start start 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.450+0000 7f932c5ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932c5ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9324079e20 con 0x7f9324107210 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932a356700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932a356700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37730/0 (socket says 172.21.15.98:37730) 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932a356700 1 -- 172.21.15.98:0/1858666834 learned_addr learned my addr 172.21.15.98:0/1858666834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:46.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932a356700 1 -- 172.21.15.98:0/1858666834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9324109b40 con 0x7f9324107210 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f932a356700 1 --2- 172.21.15.98:0/1858666834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93200098a0 tx=0x7f9320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e866f04fdd3334 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f9329354700 1 -- 172.21.15.98:0/1858666834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9320004160 con 0x7f9324107210 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f9329354700 1 -- 172.21.15.98:0/1858666834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f9320004300 con 0x7f9324107210 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.451+0000 7f9329354700 1 -- 172.21.15.98:0/1858666834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93200119b0 con 0x7f9324107210 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 -- 172.21.15.98:0/1858666834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 msgr2=0x7f9324109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 --2- 172.21.15.98:0/1858666834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93200098a0 tx=0x7f9320009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:46.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 -- 172.21.15.98:0/1858666834 shutdown_connections 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 --2- 172.21.15.98:0/1858666834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f9324109600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 -- 172.21.15.98:0/1858666834 >> 172.21.15.98:0/1858666834 conn(0x7f9324100ba0 msgr2=0x7f9324103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 -- 172.21.15.98:0/1858666834 shutdown_connections 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.452+0000 7f932c5ba700 1 -- 172.21.15.98:0/1858666834 wait complete. 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932c5ba700 1 Processor -- start 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932c5ba700 1 -- start start 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932c5ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932c5ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f932410aa50 con 0x7f9324107210 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932a356700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932a356700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37744/0 (socket says 172.21.15.98:37744) 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932a356700 1 -- 172.21.15.98:0/910877545 learned_addr learned my addr 172.21.15.98:0/910877545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932a356700 1 -- 172.21.15.98:0/910877545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932419abe0 con 0x7f9324107210 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932a356700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9320011670 tx=0x7f9320003e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:46.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93200038b0 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f93200248a0 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932419ae70 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.453+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f932001a670 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.454+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932419db60 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.454+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f9320024a50 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.454+0000 7f931b7fe700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.454+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9324063850 con 0x7f9324107210 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f931b7fe700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f931b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f931b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:46.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f931b7fe700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 0x7f931003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f932004d3d0 con 0x7f9324107210 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f9329b55700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 0x7f931003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.455+0000 7f9329b55700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 0x7f931003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9314006fd0 tx=0x7f9314006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.460+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f932002d910 con 0x7f9324107210 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.691+0000 7f932c5ba700 20 mgrc start_command cmd: [{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.691+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9324101f20 con 0x7f9310039050 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.696+0000 7f931b7fe700 1 -- 172.21.15.98:0/910877545 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9324101f20 con 0x7f9310039050 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.696+0000 7f931b7fe700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:46.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 10 mgrc shutdown 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 msgr2=0x7f931003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 0x7f931003b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9314006fd0 tx=0x7f9314006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 msgr2=0x7f932419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9320011670 tx=0x7f9320003e40 comp rx=0 tx=0).stop 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 shutdown_connections 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f9310039050 0x7f931003b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 --2- 172.21.15.98:0/910877545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9324107210 0x7f932419a6a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 >> 172.21.15.98:0/910877545 conn(0x7f9324100ba0 msgr2=0x7f9324101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 shutdown_connections 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:45.698+0000 7f932c5ba700 1 -- 172.21.15.98:0/910877545 wait complete. 2024-04-26T04:17:46.381 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-04-26T04:17:46.945 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:46 smithi098 ceph-mon[27184]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:46.945 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:46 smithi098 ceph-mon[27184]: Saving service mon spec with placement count:5 2024-04-26T04:17:46.945 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:46 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f88302a6700 1 Processor -- start 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f88302a6700 1 -- start start 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f88302a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f88302a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8828106570 con 0x7f8828105b80 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f882e042700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f882e042700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37760/0 (socket says 172.21.15.98:37760) 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.778+0000 7f882e042700 1 -- 172.21.15.98:0/1258313822 learned_addr learned my addr 172.21.15.98:0/1258313822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f882e042700 1 -- 172.21.15.98:0/1258313822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88281068a0 con 0x7f8828105b80 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f882e042700 1 --2- 172.21.15.98:0/1258313822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f88240098a0 tx=0x7f8824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87916ce69ac5c4bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f882d040700 1 -- 172.21.15.98:0/1258313822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824004160 con 0x7f8828105b80 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f882d040700 1 -- 172.21.15.98:0/1258313822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f8824004300 con 0x7f8828105b80 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f882d040700 1 -- 172.21.15.98:0/1258313822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88240119b0 con 0x7f8828105b80 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f88302a6700 1 -- 172.21.15.98:0/1258313822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 msgr2=0x7f8828105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.779+0000 7f88302a6700 1 --2- 172.21.15.98:0/1258313822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f88240098a0 tx=0x7f8824009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- 172.21.15.98:0/1258313822 shutdown_connections 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 --2- 172.21.15.98:0/1258313822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f8828105fa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- 172.21.15.98:0/1258313822 >> 172.21.15.98:0/1258313822 conn(0x7f8828100380 msgr2=0x7f88281027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- 172.21.15.98:0/1258313822 shutdown_connections 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- 172.21.15.98:0/1258313822 wait complete. 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 Processor -- start 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- start start 2024-04-26T04:17:48.180 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f88302a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8828109ad0 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.780+0000 7f882e042700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f882e042700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37768/0 (socket says 172.21.15.98:37768) 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f882e042700 1 -- 172.21.15.98:0/2971179018 learned_addr learned my addr 172.21.15.98:0/2971179018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f882e042700 1 -- 172.21.15.98:0/2971179018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8828190d20 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f882e042700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8824011670 tx=0x7f8824011820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88240038b0 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f882401a8a0 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882818ef10 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8824019550 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.781+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882818f450 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f882401a430 con 0x7f8828105b80 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:48.181 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 0x7f881403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8828109c50 con 0x7f8828105b80 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f882404d680 con 0x7f8828105b80 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.782+0000 7f882d841700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 0x7f881403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.783+0000 7f882d841700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 0x7f881403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8818006fd0 tx=0x7f8818006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.787+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f882401aa50 con 0x7f8828105b80 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.998+0000 7f88302a6700 20 mgrc start_command cmd: [{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:17:48.182 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:46.998+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f8828063850 con 0x7f88140390a0 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.005+0000 7f881f7fe700 1 -- 172.21.15.98:0/2971179018 <== mgr.14118 v2:172.21.15.98:6800/142088859 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8828063850 con 0x7f88140390a0 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.005+0000 7f881f7fe700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 10 mgrc shutdown 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 msgr2=0x7f881403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 0x7f881403b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8818006fd0 tx=0x7f8818006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 msgr2=0x7f88281907e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8824011670 tx=0x7f8824011820 comp rx=0 tx=0).stop 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 shutdown_connections 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f88140390a0 0x7f881403b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 --2- 172.21.15.98:0/2971179018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8828105b80 0x7f88281907e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 >> 172.21.15.98:0/2971179018 conn(0x7f8828100380 msgr2=0x7f8828100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 shutdown_connections 2024-04-26T04:17:48.183 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:47.007+0000 7f88302a6700 1 -- 172.21.15.98:0/2971179018 wait complete. 2024-04-26T04:17:48.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:48 smithi098 ceph-mon[27184]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:17:48.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:48 smithi098 ceph-mon[27184]: Saving service mgr spec with placement count:2 2024-04-26T04:17:48.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:48 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:48.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:48 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.649+0000 7fb14b538700 1 Processor -- start 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.649+0000 7fb14b538700 1 -- start start 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb14b538700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb14b538700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb144079e20 con 0x7fb144107210 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb1492d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb1492d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46816/0 (socket says 172.21.15.98:46816) 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb1492d4700 1 -- 172.21.15.98:0/805327620 learned_addr learned my addr 172.21.15.98:0/805327620 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:49.419 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb1492d4700 1 -- 172.21.15.98:0/805327620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb144109b40 con 0x7fb144107210 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb1492d4700 1 --2- 172.21.15.98:0/805327620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb1340098a0 tx=0x7fb134009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a997966997ee4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb13bfff700 1 -- 172.21.15.98:0/805327620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb134004160 con 0x7fb144107210 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb13bfff700 1 -- 172.21.15.98:0/805327620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb134004300 con 0x7fb144107210 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.650+0000 7fb13bfff700 1 -- 172.21.15.98:0/805327620 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb134011960 con 0x7fb144107210 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 -- 172.21.15.98:0/805327620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 msgr2=0x7fb144109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 --2- 172.21.15.98:0/805327620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb1340098a0 tx=0x7fb134009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 -- 172.21.15.98:0/805327620 shutdown_connections 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 --2- 172.21.15.98:0/805327620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb144109600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 -- 172.21.15.98:0/805327620 >> 172.21.15.98:0/805327620 conn(0x7fb144100ba0 msgr2=0x7fb144103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 -- 172.21.15.98:0/805327620 shutdown_connections 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.651+0000 7fb14b538700 1 -- 172.21.15.98:0/805327620 wait complete. 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 Processor -- start 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 -- start start 2024-04-26T04:17:49.420 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb14410aa90 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb1492d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb1492d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46822/0 (socket says 172.21.15.98:46822) 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb1492d4700 1 -- 172.21.15.98:0/2682265819 learned_addr learned my addr 172.21.15.98:0/2682265819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb1492d4700 1 -- 172.21.15.98:0/2682265819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14419abe0 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb1492d4700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb134003ed0 tx=0x7fb134004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1340038b0 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb13401a8c0 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14419ae70 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb134019640 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.652+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb14419db60 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb13401a430 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb144063850 con 0x7fb144107210 2024-04-26T04:17:49.421 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 0x7fb13003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.653+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb13404d650 con 0x7fb144107210 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.654+0000 7fb148ad3700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 0x7fb13003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.654+0000 7fb148ad3700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 0x7fb13003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb140006fd0 tx=0x7fb140006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.658+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb134019da0 con 0x7fb144107210 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.866+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fb14419df40 con 0x7fb144107210 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.872+0000 7fb13a7fc700 1 -- 172.21.15.98:0/2682265819 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fb134019880 con 0x7fb144107210 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.874+0000 7fb14b538700 10 mgrc shutdown 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.874+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 msgr2=0x7fb13003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.874+0000 7fb14b538700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 0x7fb13003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb140006fd0 tx=0x7fb140006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.874+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 msgr2=0x7fb14419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:49.422 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.874+0000 7fb14b538700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb134003ed0 tx=0x7fb134004290 comp rx=0 tx=0).stop 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 shutdown_connections 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7fb130039050 0x7fb13003b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 --2- 172.21.15.98:0/2682265819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb144107210 0x7fb14419a6a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 >> 172.21.15.98:0/2682265819 conn(0x7fb144100ba0 msgr2=0x7fb1441017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 shutdown_connections 2024-04-26T04:17:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:48.875+0000 7fb14b538700 1 -- 172.21.15.98:0/2682265819 wait complete. 2024-04-26T04:17:49.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:49 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:49.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:49 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2682265819' entity='client.admin' 2024-04-26T04:17:51.162 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:51 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2674157117' entity='client.admin' 2024-04-26T04:17:51.165 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.847+0000 7f8ad8637700 1 Processor -- start 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.847+0000 7f8ad8637700 1 -- start start 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad8637700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad8637700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0107180 con 0x7f8ad0106790 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad63d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad63d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46834/0 (socket says 172.21.15.98:46834) 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad63d3700 1 -- 172.21.15.98:0/453808271 learned_addr learned my addr 172.21.15.98:0/453808271 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad63d3700 1 -- 172.21.15.98:0/453808271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad0107990 con 0x7f8ad0106790 2024-04-26T04:17:51.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad63d3700 1 --2- 172.21.15.98:0/453808271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00098a0 tx=0x7f8ac0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f493acca47a3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad53d1700 1 -- 172.21.15.98:0/453808271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ac0004160 con 0x7f8ad0106790 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad53d1700 1 -- 172.21.15.98:0/453808271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ac0004300 con 0x7f8ad0106790 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.848+0000 7f8ad53d1700 1 -- 172.21.15.98:0/453808271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ac0011960 con 0x7f8ad0106790 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 -- 172.21.15.98:0/453808271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 msgr2=0x7f8ad0106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 --2- 172.21.15.98:0/453808271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00098a0 tx=0x7f8ac0009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 -- 172.21.15.98:0/453808271 shutdown_connections 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 --2- 172.21.15.98:0/453808271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad0106bb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 -- 172.21.15.98:0/453808271 >> 172.21.15.98:0/453808271 conn(0x7f8ad0101d50 msgr2=0x7f8ad0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 -- 172.21.15.98:0/453808271 shutdown_connections 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.849+0000 7f8ad8637700 1 -- 172.21.15.98:0/453808271 wait complete. 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad8637700 1 Processor -- start 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad8637700 1 -- start start 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad8637700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:51.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad8637700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad010abc0 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad63d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad63d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46848/0 (socket says 172.21.15.98:46848) 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad63d3700 1 -- 172.21.15.98:0/2674157117 learned_addr learned my addr 172.21.15.98:0/2674157117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad63d3700 1 -- 172.21.15.98:0/2674157117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad0079160 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ad63d3700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0006af0 tx=0x7f8ac0003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.850+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ac0011700 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ac00118a0 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ac001a5c0 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad00793f0 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad00798d0 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8ac001a870 con 0x7f8ad0106790 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.851+0000 7f8ac77fe700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ac77fe700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ac77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:51.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ac77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ac77fe700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 0x7f8abc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8ac004d610 con 0x7f8ad0106790 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ad5bd2700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 0x7f8abc03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad010ad40 con 0x7f8ad0106790 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.852+0000 7f8ad5bd2700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 0x7f8abc03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8acc006fd0 tx=0x7f8acc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:49.861+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ac0019c70 con 0x7f8ad0106790 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.097+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f8ad0079b50 con 0x7f8ad0106790 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.101+0000 7f8ac77fe700 1 -- 172.21.15.98:0/2674157117 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f8ac002cc00 con 0x7f8ad0106790 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.103+0000 7f8ad8637700 10 mgrc shutdown 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.103+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 msgr2=0x7f8abc03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 0x7f8abc03b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8acc006fd0 tx=0x7f8acc006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 msgr2=0x7f8ad007ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0006af0 tx=0x7f8ac0003750 comp rx=0 tx=0).stop 2024-04-26T04:17:51.174 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 shutdown_connections 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f8abc039050 0x7f8abc03b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 --2- 172.21.15.98:0/2674157117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ad0106790 0x7f8ad007ac10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 >> 172.21.15.98:0/2674157117 conn(0x7f8ad0101d50 msgr2=0x7f8ad01027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 shutdown_connections 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:50.104+0000 7f8ad8637700 1 -- 172.21.15.98:0/2674157117 wait complete. 2024-04-26T04:17:51.175 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-04-26T04:17:52.978 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:52 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: ignoring --setuser ceph since I am not root 2024-04-26T04:17:52.978 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:52 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: ignoring --setgroup ceph since I am not root 2024-04-26T04:17:52.978 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:52 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:52.873+0000 7f834ea32200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T04:17:52.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:52 smithi098 ceph-mon[27184]: from='mgr.14118 172.21.15.98:0/3167560791' entity='mgr.a' 2024-04-26T04:17:52.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:52 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/349496407' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T04:17:53.291 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:53.059+0000 7f834ea32200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:17:53.291 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:53.159+0000 7f834ea32200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T04:17:53.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9f19a700 1 Processor -- start 2024-04-26T04:17:53.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9f19a700 1 -- start start 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9f19a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9f19a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98079e40 con 0x7f2b981072d0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9cf36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9cf36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46852/0 (socket says 172.21.15.98:46852) 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9cf36700 1 -- 172.21.15.98:0/2163336624 learned_addr learned my addr 172.21.15.98:0/2163336624 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9cf36700 1 -- 172.21.15.98:0/2163336624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b98109c00 con 0x7f2b981072d0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.616+0000 7f2b9cf36700 1 --2- 172.21.15.98:0/2163336624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098a0 tx=0x7f2b8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf88552f61387a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b977fe700 1 -- 172.21.15.98:0/2163336624 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b8c004160 con 0x7f2b981072d0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b977fe700 1 -- 172.21.15.98:0/2163336624 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2b8c004300 con 0x7f2b981072d0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b977fe700 1 -- 172.21.15.98:0/2163336624 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b8c011960 con 0x7f2b981072d0 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 -- 172.21.15.98:0/2163336624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 msgr2=0x7f2b981096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/2163336624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098a0 tx=0x7f2b8c009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:53.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 -- 172.21.15.98:0/2163336624 shutdown_connections 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/2163336624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b981096c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 -- 172.21.15.98:0/2163336624 >> 172.21.15.98:0/2163336624 conn(0x7f2b98100d00 msgr2=0x7f2b98103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 -- 172.21.15.98:0/2163336624 shutdown_connections 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.617+0000 7f2b9f19a700 1 -- 172.21.15.98:0/2163336624 wait complete. 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9f19a700 1 Processor -- start 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9f19a700 1 -- start start 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9f19a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9f19a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9810aba0 con 0x7f2b981072d0 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9cf36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9cf36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46854/0 (socket says 172.21.15.98:46854) 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9cf36700 1 -- 172.21.15.98:0/349496407 learned_addr learned my addr 172.21.15.98:0/349496407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.618+0000 7f2b9cf36700 1 -- 172.21.15.98:0/349496407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9819abd0 con 0x7f2b981072d0 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b9cf36700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c006af0 tx=0x7f2b8c003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:53.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b8c0039a0 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2b8c011910 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9819ae60 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b8c01a5c0 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.619+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9819db50 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2b8c01a870 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 20 mgrc handle_mgr_map mgrmap(e 8) 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 4 mgrc handle_mgr_map Got map version 8 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 0x7f2b8803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b8c04d3d0 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b98063850 con 0x7f2b981072d0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b97fff700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 0x7f2b8803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:53.307 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.620+0000 7f2b97fff700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 0x7f2b8803b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.625+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2b8c020240 con 0x7f2b981072d0 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:51.901+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f2b9819ddc0 con 0x7f2b981072d0 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.695+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2b8c00ba20 con 0x7f2b981072d0 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.697+0000 7f2b95ffb700 1 -- 172.21.15.98:0/349496407 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f2b8c02d580 con 0x7f2b981072d0 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.697+0000 7f2b95ffb700 20 mgrc handle_mgr_map mgrmap(e 9) 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.697+0000 7f2b95ffb700 4 mgrc handle_mgr_map Got map version 9 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.697+0000 7f2b95ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 10 mgrc shutdown 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 msgr2=0x7f2b8803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 0x7f2b8803b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b84006fd0 tx=0x7f2b84006e40 comp rx=0 tx=0).stop 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 msgr2=0x7f2b9819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c006af0 tx=0x7f2b8c003750 comp rx=0 tx=0).stop 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 shutdown_connections 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f2b88039050 0x7f2b8803b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 --2- 172.21.15.98:0/349496407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b981072d0 0x7f2b9819a690 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:53.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 >> 172.21.15.98:0/349496407 conn(0x7f2b98100d00 msgr2=0x7f2b98101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:53.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 shutdown_connections 2024-04-26T04:17:53.309 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:52.698+0000 7f2b9f19a700 1 -- 172.21.15.98:0/349496407 wait complete. 2024-04-26T04:17:53.557 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:53.471+0000 7f834ea32200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T04:17:53.557 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:53.556+0000 7f834ea32200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T04:17:53.810 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/349496407' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T04:17:53.810 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:53 smithi098 ceph-mon[27184]: mgrmap e9: a(active, since 19s) 2024-04-26T04:17:54.703 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "a", 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f703ad5a700 1 Processor -- start 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f703ad5a700 1 -- start start 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f703ad5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f703ad5a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7034105430 con 0x7f7034104a40 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f7038af6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f7038af6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46882/0 (socket says 172.21.15.98:46882) 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.809+0000 7f7038af6700 1 -- 172.21.15.98:0/667242834 learned_addr learned my addr 172.21.15.98:0/667242834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.810+0000 7f7038af6700 1 -- 172.21.15.98:0/667242834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70341079d0 con 0x7f7034104a40 2024-04-26T04:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.810+0000 7f7038af6700 1 --2- 172.21.15.98:0/667242834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70280098a0 tx=0x7f7028009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7207595ebb45ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.810+0000 7f70337fe700 1 -- 172.21.15.98:0/667242834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7028004160 con 0x7f7034104a40 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.810+0000 7f70337fe700 1 -- 172.21.15.98:0/667242834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7028004300 con 0x7f7034104a40 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.810+0000 7f70337fe700 1 -- 172.21.15.98:0/667242834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7028011960 con 0x7f7034104a40 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 -- 172.21.15.98:0/667242834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 msgr2=0x7f7034104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 --2- 172.21.15.98:0/667242834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70280098a0 tx=0x7f7028009bb0 comp rx=0 tx=0).stop 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 -- 172.21.15.98:0/667242834 shutdown_connections 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 --2- 172.21.15.98:0/667242834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034104e60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 -- 172.21.15.98:0/667242834 >> 172.21.15.98:0/667242834 conn(0x7f7034100330 msgr2=0x7f70341027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 -- 172.21.15.98:0/667242834 shutdown_connections 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 -- 172.21.15.98:0/667242834 wait complete. 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.811+0000 7f703ad5a700 1 Processor -- start 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f703ad5a700 1 -- start start 2024-04-26T04:17:54.705 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f703ad5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f703ad5a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f703410ac00 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7038af6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7038af6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46898/0 (socket says 172.21.15.98:46898) 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7038af6700 1 -- 172.21.15.98:0/2122621300 learned_addr learned my addr 172.21.15.98:0/2122621300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7038af6700 1 -- 172.21.15.98:0/2122621300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7034107790 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7038af6700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7028003ed0 tx=0x7f7028004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7028011700 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f70280118a0 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7028019550 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.812+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7034105980 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.813+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7034105e00 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f7028019800 con 0x7f7034104a40 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 20 mgrc handle_mgr_map mgrmap(e 9) 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 4 mgrc handle_mgr_map Got map version 9 2024-04-26T04:17:54.706 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 0x7f702403b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f702804d710 con 0x7f7034104a40 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7033fff700 1 -- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 msgr2=0x7f702403b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f7033fff700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 0x7f702403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.814+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703410ad80 con 0x7f7034104a40 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:53.819+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f702801aa30 con 0x7f7034104a40 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.015+0000 7f7033fff700 1 -- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 msgr2=0x7f702403b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.015+0000 7f7033fff700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 0x7f702403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.115+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7034106300 con 0x7f7034104a40 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.116+0000 7f7031ffb700 1 -- 172.21.15.98:0/2122621300 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f702802d960 con 0x7f7034104a40 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 10 mgrc shutdown 2024-04-26T04:17:54.707 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 msgr2=0x7f702403b570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 0x7f702403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 msgr2=0x7f7034107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7028003ed0 tx=0x7f7028004290 comp rx=0 tx=0).stop 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 shutdown_connections 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f70240390b0 0x7f702403b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 --2- 172.21.15.98:0/2122621300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7034104a40 0x7f7034107250 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 >> 172.21.15.98:0/2122621300 conn(0x7f7034100330 msgr2=0x7f7034100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 shutdown_connections 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:54.118+0000 7f703ad5a700 1 -- 172.21.15.98:0/2122621300 wait complete. 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-26T04:17:54.708 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-04-26T04:17:55.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:54 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2122621300' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T04:17:55.107 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:54 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:54.766+0000 7f834ea32200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T04:17:55.107 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:54 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:54.941+0000 7f834ea32200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:55.291 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:55 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:55.113+0000 7f834ea32200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T04:17:55.698 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:55 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:55.404+0000 7f834ea32200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T04:17:55.698 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:55 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:55.495+0000 7f834ea32200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T04:17:55.698 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:55 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:55.696+0000 7f834ea32200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T04:17:56.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:55 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:55.889+0000 7f834ea32200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:17:56.791 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:56 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:56.497+0000 7f834ea32200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T04:17:57.212 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:56 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:56.916+0000 7f834ea32200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T04:17:57.212 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:57 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:57.012+0000 7f834ea32200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T04:17:57.212 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:57 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:57.211+0000 7f834ea32200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T04:17:58.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:57 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:57.734+0000 7f834ea32200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T04:17:58.041 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:57 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:57.825+0000 7f834ea32200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T04:17:58.411 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:58 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:58.066+0000 7f834ea32200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T04:17:58.411 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:58 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:58.199+0000 7f834ea32200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T04:17:58.411 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:58 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:58.294+0000 7f834ea32200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T04:17:58.411 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:58 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:58.410+0000 7f834ea32200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T04:17:58.791 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:58 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:58.502+0000 7f834ea32200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T04:17:59.579 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:59.308+0000 7f834ea32200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T04:17:59.579 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:17:59.524+0000 7f834ea32200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T04:18:00.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: Active manager daemon a restarted 2024-04-26T04:18:00.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: Activating manager daemon a 2024-04-26T04:18:00.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T04:18:00.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: mgrmap e10: a(active, starting, since 0.00348723s) 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mgr metadata", "who": "a", "id": "a"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: Manager daemon a is now available 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/mirror_snapshot_schedule"}]: dispatch 2024-04-26T04:18:00.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:17:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/a/trash_purge_schedule"}]: dispatch 2024-04-26T04:18:01.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-26T04:18:01.524 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.131+0000 7f3a5a87f700 1 Processor -- start 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.131+0000 7f3a5a87f700 1 -- start start 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.131+0000 7f3a5a87f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.131+0000 7f3a5a87f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54106060 con 0x7f3a54105670 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46904/0 (socket says 172.21.15.98:46904) 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a53fff700 1 -- 172.21.15.98:0/552677785 learned_addr learned my addr 172.21.15.98:0/552677785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a53fff700 1 -- 172.21.15.98:0/552677785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a54106870 con 0x7f3a54105670 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a53fff700 1 --2- 172.21.15.98:0/552677785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a440098a0 tx=0x7f3a44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddc44295890648f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a52ffd700 1 -- 172.21.15.98:0/552677785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44004160 con 0x7f3a54105670 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a52ffd700 1 -- 172.21.15.98:0/552677785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a44004300 con 0x7f3a54105670 2024-04-26T04:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.132+0000 7f3a52ffd700 1 -- 172.21.15.98:0/552677785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44011960 con 0x7f3a54105670 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- 172.21.15.98:0/552677785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 msgr2=0x7f3a54105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/552677785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a440098a0 tx=0x7f3a44009bb0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- 172.21.15.98:0/552677785 shutdown_connections 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/552677785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a54105a90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- 172.21.15.98:0/552677785 >> 172.21.15.98:0/552677785 conn(0x7f3a54100c10 msgr2=0x7f3a54103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- 172.21.15.98:0/552677785 shutdown_connections 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- 172.21.15.98:0/552677785 wait complete. 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 Processor -- start 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.133+0000 7f3a5a87f700 1 -- start start 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a5a87f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a5a87f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54109aa0 con 0x7f3a54105670 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:01.526 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46910/0 (socket says 172.21.15.98:46910) 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a53fff700 1 -- 172.21.15.98:0/785666448 learned_addr learned my addr 172.21.15.98:0/785666448 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a53fff700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5419ac40 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a53fff700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a44003ed0 tx=0x7f3a44004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44011700 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a440118a0 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5419aed0 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a44019550 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.134+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a5419dbc0 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a5a87f700 20 mgrc start_tell_command target: cmd: [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a5a87f700 5 mgrc start_tell_command no mgr session (no running mgr daemon?), or not active mgr, waiting 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f3a44019850 con 0x7f3a54105670 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 20 mgrc handle_mgr_map mgrmap(e 9) 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 4 mgrc handle_mgr_map Got map version 9 2024-04-26T04:18:01.527 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 10 mgrc reconnect resending 0 (tell) 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a3c03bc20 con 0x7f3a3c039050 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3a4404d5a0 con 0x7f3a54105670 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a537fe700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.135+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.336+0000 7f3a537fe700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.336+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.737+0000 7f3a537fe700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:55.737+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:56.538+0000 7f3a537fe700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:56.538+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:58.139+0000 7f3a537fe700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.528 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:58.139+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f3a44019be0 con 0x7f3a54105670 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 20 mgrc handle_mgr_map mgrmap(e 10) 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 4 mgrc handle_mgr_map Got map version 10 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 4 mgrc handle_mgr_map Active mgr is now 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 4 mgrc reconnect Terminating session with v2:172.21.15.98:6800/142088859 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 msgr2=0x7f3a3c03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:17:59.528+0000 7f3a517fa700 4 mgrc reconnect No active mgr available yet 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f3a4404f6d0 con 0x7f3a54105670 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 20 mgrc handle_mgr_map mgrmap(e 11) 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 4 mgrc handle_mgr_map Got map version 11 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 0x7f3a3c03ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:01.529 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 10 mgrc reconnect resending 0 (tell) 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a44008ea0 con 0x7f3a3c03cb10 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.529+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 0x7f3a3c03ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.530+0000 7f3a537fe700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 0x7f3a3c03ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3a48003a10 tx=0x7f3a480092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.531+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f3a44008ea0 con 0x7f3a3c03cb10 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.531+0000 7f3a517fa700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.537+0000 7f3a5a87f700 20 mgrc start_tell_command target: cmd: [{"prefix": "mgr_status"}] 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.537+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3a5419b3b0 con 0x7f3a3c03cb10 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.537+0000 7f3a517fa700 1 -- 172.21.15.98:0/785666448 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f3a5419b3b0 con 0x7f3a3c03cb10 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.537+0000 7f3a517fa700 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 10 mgrc shutdown 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 msgr2=0x7f3a3c03ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 0x7f3a3c03ef00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3a48003a10 tx=0x7f3a480092b0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 msgr2=0x7f3a5419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:01.530 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a44003ed0 tx=0x7f3a44004290 comp rx=0 tx=0).stop 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 shutdown_connections 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3a3c03cb10 0x7f3a3c03ef00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a54105670 0x7f3a5419a700 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 --2- 172.21.15.98:0/785666448 >> [v2:172.21.15.98:6800/142088859,v1:172.21.15.98:6801/142088859] conn(0x7f3a3c039050 0x7f3a3c03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 >> 172.21.15.98:0/785666448 conn(0x7f3a54100c10 msgr2=0x7f3a54101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 shutdown_connections 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:00.538+0000 7f3a5a87f700 1 -- 172.21.15.98:0/785666448 wait complete. 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-04-26T04:18:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: [26/Apr/2024:04:18:00] ENGINE Bus STARTING 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: [26/Apr/2024:04:18:00] ENGINE Serving on http://172.21.15.98:8765 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: [26/Apr/2024:04:18:00] ENGINE Serving on https://172.21.15.98:7150 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: [26/Apr/2024:04:18:00] ENGINE Bus STARTED 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: mgrmap e11: a(active, since 1.00595s) 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T04:18:01.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:02.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T04:18:02.904 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f988d8f1700 1 Processor -- start 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f988d8f1700 1 -- start start 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f988d8f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f988d8f1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9888106070 con 0x7f9888105680 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f9886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f9886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53856/0 (socket says 172.21.15.98:53856) 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f9886ffd700 1 -- 172.21.15.98:0/2561888425 learned_addr learned my addr 172.21.15.98:0/2561888425 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.001+0000 7f9886ffd700 1 -- 172.21.15.98:0/2561888425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9888106880 con 0x7f9888105680 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f9886ffd700 1 --2- 172.21.15.98:0/2561888425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98780098a0 tx=0x7f9878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16217835063036ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f9885ffb700 1 -- 172.21.15.98:0/2561888425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9878004160 con 0x7f9888105680 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f9885ffb700 1 -- 172.21.15.98:0/2561888425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9878004300 con 0x7f9888105680 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f9885ffb700 1 -- 172.21.15.98:0/2561888425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9878011960 con 0x7f9888105680 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f988d8f1700 1 -- 172.21.15.98:0/2561888425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 msgr2=0x7f9888105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:02.905 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.002+0000 7f988d8f1700 1 --2- 172.21.15.98:0/2561888425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98780098a0 tx=0x7f9878009bb0 comp rx=0 tx=0).stop 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- 172.21.15.98:0/2561888425 shutdown_connections 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 --2- 172.21.15.98:0/2561888425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888105aa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- 172.21.15.98:0/2561888425 >> 172.21.15.98:0/2561888425 conn(0x7f9888100c60 msgr2=0x7f9888103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- 172.21.15.98:0/2561888425 shutdown_connections 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- 172.21.15.98:0/2561888425 wait complete. 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 Processor -- start 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- start start 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.003+0000 7f988d8f1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9888109ab0 con 0x7f9888105680 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f9886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f9886ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53872/0 (socket says 172.21.15.98:53872) 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f9886ffd700 1 -- 172.21.15.98:0/3292168142 learned_addr learned my addr 172.21.15.98:0/3292168142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f9886ffd700 1 -- 172.21.15.98:0/3292168142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988818eea0 con 0x7f9888105680 2024-04-26T04:18:02.906 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f9886ffd700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9878003ed0 tx=0x7f9878004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9878011a20 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98780038b0 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988818f130 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f987801a5a0 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.004+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988818f610 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.005+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9888109c30 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.005+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f987801a8a0 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.005+0000 7f988c8ef700 20 mgrc handle_mgr_map mgrmap(e 11) 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f988c8ef700 4 mgrc handle_mgr_map Got map version 11 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f988c8ef700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f988c8ef700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f988c8ef700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 0x7f987003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f987804d570 con 0x7f9888105680 2024-04-26T04:18:02.907 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f98867fc700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 0x7f987003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.006+0000 7f98867fc700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 0x7f987003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.011+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9878024b10 con 0x7f9888105680 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.131+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f987802d770 con 0x7f9888105680 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.131+0000 7f988c8ef700 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.132+0000 7f988c8ef700 4 mgrc handle_mgr_map Got map version 12 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.132+0000 7f988c8ef700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.223+0000 7f988d8f1700 20 mgrc start_command cmd: [{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.223+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f988818f890 con 0x7f9870038f80 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.347+0000 7f988c8ef700 1 -- 172.21.15.98:0/3292168142 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f988818f890 con 0x7f9870038f80 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.347+0000 7f988c8ef700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 10 mgrc shutdown 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 msgr2=0x7f987003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 0x7f987003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f987c006fd0 tx=0x7f987c006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:02.908 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 msgr2=0x7f9888190950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9878003ed0 tx=0x7f9878004290 comp rx=0 tx=0).stop 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 shutdown_connections 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9870038f80 0x7f987003b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 --2- 172.21.15.98:0/3292168142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9888105680 0x7f9888190950 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 >> 172.21.15.98:0/3292168142 conn(0x7f9888100c60 msgr2=0x7f9888101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 shutdown_connections 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:02.350+0000 7f988d8f1700 1 -- 172.21.15.98:0/3292168142 wait complete. 2024-04-26T04:18:02.909 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-04-26T04:18:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:03 smithi098 ceph-mon[27184]: mgrmap e12: a(active, since 2s) 2024-04-26T04:18:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:03 smithi098 ceph-mon[27184]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:03 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:03 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:03 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:04.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:04 smithi098 ceph-mon[27184]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:04.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$fZE3pV8Im7VaR2ebfsBHJ.nMPFizE7F9oKfjZwPbutKt5INFxeSbq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714105083, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.343+0000 7fab6332d700 1 Processor -- start 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.343+0000 7fab6332d700 1 -- start start 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.343+0000 7fab6332d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.343+0000 7fab6332d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c105430 con 0x7fab5c104a40 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab610c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab610c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53880/0 (socket says 172.21.15.98:53880) 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab610c9700 1 -- 172.21.15.98:0/4232467324 learned_addr learned my addr 172.21.15.98:0/4232467324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab610c9700 1 -- 172.21.15.98:0/4232467324 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c1079d0 con 0x7fab5c104a40 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab610c9700 1 --2- 172.21.15.98:0/4232467324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0098a0 tx=0x7fab4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc7c53f81b150a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab53fff700 1 -- 172.21.15.98:0/4232467324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab4c004160 con 0x7fab5c104a40 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab53fff700 1 -- 172.21.15.98:0/4232467324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fab4c004300 con 0x7fab5c104a40 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.344+0000 7fab53fff700 1 -- 172.21.15.98:0/4232467324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab4c011960 con 0x7fab5c104a40 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 -- 172.21.15.98:0/4232467324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 msgr2=0x7fab5c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 --2- 172.21.15.98:0/4232467324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab4c0098a0 tx=0x7fab4c009bb0 comp rx=0 tx=0).stop 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 -- 172.21.15.98:0/4232467324 shutdown_connections 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 --2- 172.21.15.98:0/4232467324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c104e60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 -- 172.21.15.98:0/4232467324 >> 172.21.15.98:0/4232467324 conn(0x7fab5c100330 msgr2=0x7fab5c1027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 -- 172.21.15.98:0/4232467324 shutdown_connections 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 -- 172.21.15.98:0/4232467324 wait complete. 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.345+0000 7fab6332d700 1 Processor -- start 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab6332d700 1 -- start start 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab6332d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab6332d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c10ac00 con 0x7fab5c104a40 2024-04-26T04:18:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab610c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab610c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53884/0 (socket says 172.21.15.98:53884) 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab610c9700 1 -- 172.21.15.98:0/18358419 learned_addr learned my addr 172.21.15.98:0/18358419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab610c9700 1 -- 172.21.15.98:0/18358419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c107790 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab610c9700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab4c003ed0 tx=0x7fab4c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab4c011700 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c105980 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fab4c0118a0 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab4c019550 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.346+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c105ec0 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fab4c019870 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c10ad80 con 0x7fab5c104a40 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 4 mgrc handle_mgr_map Got map version 12 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:04.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.347+0000 7fab527fc700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 0x7fab4803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.348+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fab4c04d680 con 0x7fab5c104a40 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.348+0000 7fab608c8700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 0x7fab4803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.348+0000 7fab608c8700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 0x7fab4803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.352+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab4c01aa30 con 0x7fab5c104a40 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.581+0000 7fab6332d700 20 mgrc start_command cmd: [{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.581+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fab5c063850 con 0x7fab48039050 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.805+0000 7fab527fc700 1 -- 172.21.15.98:0/18358419 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fab5c063850 con 0x7fab48039050 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.805+0000 7fab527fc700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 10 mgrc shutdown 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 msgr2=0x7fab4803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 0x7fab4803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab58006fd0 tx=0x7fab58006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 msgr2=0x7fab5c107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab4c003ed0 tx=0x7fab4c004290 comp rx=0 tx=0).stop 2024-04-26T04:18:04.761 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 shutdown_connections 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fab48039050 0x7fab4803b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 --2- 172.21.15.98:0/18358419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab5c104a40 0x7fab5c107250 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.807+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 >> 172.21.15.98:0/18358419 conn(0x7fab5c100330 msgr2=0x7fab5c100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.808+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 shutdown_connections 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:03.808+0000 7fab6332d700 1 -- 172.21.15.98:0/18358419 wait complete. 2024-04-26T04:18:04.762 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-04-26T04:18:05.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:05 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2538310784' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T04:18:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T04:18:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.182+0000 7f10eb88c700 1 Processor -- start 2024-04-26T04:18:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.182+0000 7f10eb88c700 1 -- start start 2024-04-26T04:18:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.182+0000 7f10eb88c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.182+0000 7f10eb88c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10e4106050 con 0x7f10e4105660 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10e9628700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10e9628700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53900/0 (socket says 172.21.15.98:53900) 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10e9628700 1 -- 172.21.15.98:0/1180974618 learned_addr learned my addr 172.21.15.98:0/1180974618 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10e9628700 1 -- 172.21.15.98:0/1180974618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e4106860 con 0x7f10e4105660 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10e9628700 1 --2- 172.21.15.98:0/1180974618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f10e00098a0 tx=0x7f10e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7351285bdee6b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10dbfff700 1 -- 172.21.15.98:0/1180974618 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e0004160 con 0x7f10e4105660 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10dbfff700 1 -- 172.21.15.98:0/1180974618 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f10e0004300 con 0x7f10e4105660 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.183+0000 7f10dbfff700 1 -- 172.21.15.98:0/1180974618 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e0011960 con 0x7f10e4105660 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- 172.21.15.98:0/1180974618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 msgr2=0x7f10e4105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 --2- 172.21.15.98:0/1180974618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f10e00098a0 tx=0x7f10e0009bb0 comp rx=0 tx=0).stop 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- 172.21.15.98:0/1180974618 shutdown_connections 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 --2- 172.21.15.98:0/1180974618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4105a80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:06.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- 172.21.15.98:0/1180974618 >> 172.21.15.98:0/1180974618 conn(0x7f10e4100c20 msgr2=0x7f10e4103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- 172.21.15.98:0/1180974618 shutdown_connections 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- 172.21.15.98:0/1180974618 wait complete. 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 Processor -- start 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.184+0000 7f10eb88c700 1 -- start start 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10eb88c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10eb88c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10e4109a90 con 0x7f10e4105660 2024-04-26T04:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10e9628700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10e9628700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53912/0 (socket says 172.21.15.98:53912) 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10e9628700 1 -- 172.21.15.98:0/2538310784 learned_addr learned my addr 172.21.15.98:0/2538310784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10e9628700 1 -- 172.21.15.98:0/2538310784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e419d360 con 0x7f10e4105660 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10e9628700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10e0006af0 tx=0x7f10e0004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e00038b0 con 0x7f10e4105660 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f10e00248c0 con 0x7f10e4105660 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e419d530 con 0x7f10e4105660 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10e001a5c0 con 0x7f10e4105660 2024-04-26T04:18:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.185+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e419d950 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f10e001a870 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 20 mgrc handle_mgr_map mgrmap(e 12) 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 4 mgrc handle_mgr_map Got map version 12 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 0x7f10d003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e4109c10 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f10e004d700 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.186+0000 7f10e8e27700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 0x7f10d003b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.187+0000 7f10e8e27700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 0x7f10d003b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f10d4006fd0 tx=0x7f10d4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.191+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f10e0021e30 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.394+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f10e419dc40 con 0x7f10e4105660 2024-04-26T04:18:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.394+0000 7f10da7fc700 1 -- 172.21.15.98:0/2538310784 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f10e000e260 con 0x7f10e4105660 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 10 mgrc shutdown 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 msgr2=0x7f10d003b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 0x7f10d003b5c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f10d4006fd0 tx=0x7f10d4006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 msgr2=0x7f10e4079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10e0006af0 tx=0x7f10e0004290 comp rx=0 tx=0).stop 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 shutdown_connections 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f10d0039100 0x7f10d003b5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.396+0000 7f10eb88c700 1 --2- 172.21.15.98:0/2538310784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10e4105660 0x7f10e4079a40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.397+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 >> 172.21.15.98:0/2538310784 conn(0x7f10e4100c20 msgr2=0x7f10e4101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.397+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 shutdown_connections 2024-04-26T04:18:06.140 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:05.397+0000 7f10eb88c700 1 -- 172.21.15.98:0/2538310784 wait complete. 2024-04-26T04:18:06.141 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T04:18:06.141 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-26T04:18:06.147 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T04:18:06.147 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-26T04:18:06.147 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-26T04:18:06.147 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098:8443/ 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: Password: zh5u5wgpig 2024-04-26T04:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:06.150 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config directory 2024-04-26T04:18:06.150 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-04-26T04:18:07.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.592+0000 7fbda176f700 1 Processor -- start 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.592+0000 7fbda176f700 1 -- start start 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.592+0000 7fbda176f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.592+0000 7fbda176f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c079e00 con 0x7fbd9c105020 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53928/0 (socket says 172.21.15.98:53928) 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9affd700 1 -- 172.21.15.98:0/3469790129 learned_addr learned my addr 172.21.15.98:0/3469790129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9affd700 1 -- 172.21.15.98:0/3469790129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c105980 con 0x7fbd9c105020 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9affd700 1 --2- 172.21.15.98:0/3469790129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098a0 tx=0x7fbd8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d04c75675649a93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:07.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9a7fc700 1 -- 172.21.15.98:0/3469790129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c004160 con 0x7fbd9c105020 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9a7fc700 1 -- 172.21.15.98:0/3469790129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd8c004300 con 0x7fbd9c105020 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.593+0000 7fbd9a7fc700 1 -- 172.21.15.98:0/3469790129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c011960 con 0x7fbd9c105020 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 -- 172.21.15.98:0/3469790129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 msgr2=0x7fbd9c105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 --2- 172.21.15.98:0/3469790129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098a0 tx=0x7fbd8c009bb0 comp rx=0 tx=0).stop 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 -- 172.21.15.98:0/3469790129 shutdown_connections 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 --2- 172.21.15.98:0/3469790129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c105440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 -- 172.21.15.98:0/3469790129 >> 172.21.15.98:0/3469790129 conn(0x7fbd9c100ba0 msgr2=0x7fbd9c103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 -- 172.21.15.98:0/3469790129 shutdown_connections 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.594+0000 7fbda176f700 1 -- 172.21.15.98:0/3469790129 wait complete. 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbda176f700 1 Processor -- start 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbda176f700 1 -- start start 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbda176f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbda176f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9c106990 con 0x7fbd9c105020 2024-04-26T04:18:07.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbd9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbd9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53940/0 (socket says 172.21.15.98:53940) 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbd9affd700 1 -- 172.21.15.98:0/2361527411 learned_addr learned my addr 172.21.15.98:0/2361527411 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbd9affd700 1 -- 172.21.15.98:0/2361527411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9c19cde0 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.595+0000 7fbd9affd700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c003ed0 tx=0x7fbd8c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.596+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c011700 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.596+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd8c0118a0 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.596+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9c19d070 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.596+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c019550 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.596+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9c19fd60 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fbd8c01a430 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd9c063850 con 0x7fbd9c105020 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:07.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 0x7fbd8803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbd8c04d7f0 con 0x7fbd9c105020 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd927fc700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 0x7fbd8803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.597+0000 7fbd927fc700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 0x7fbd8803b5b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006fd0 tx=0x7fbd84006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.602+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd8c023e00 con 0x7fbd9c105020 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.799+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fbd9c02dc00 con 0x7fbd9c105020 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.800+0000 7fbd98ff9700 1 -- 172.21.15.98:0/2361527411 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fbd8c01a720 con 0x7fbd9c105020 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 10 mgrc shutdown 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 msgr2=0x7fbd8803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 0x7fbd8803b5b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006fd0 tx=0x7fbd84006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 msgr2=0x7fbd9c19c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:07.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c003ed0 tx=0x7fbd8c004290 comp rx=0 tx=0).stop 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 shutdown_connections 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd880390f0 0x7fbd8803b5b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 --2- 172.21.15.98:0/2361527411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd9c105020 0x7fbd9c19c8a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 >> 172.21.15.98:0/2361527411 conn(0x7fbd9c100ba0 msgr2=0x7fbd9c193480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 shutdown_connections 2024-04-26T04:18:07.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:06.802+0000 7fbda176f700 1 -- 172.21.15.98:0/2361527411 wait complete. 2024-04-26T04:18:07.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:07 smithi098 ceph-mon[27184]: mgrmap e13: a(active, since 6s) 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.774+0000 7f09aba68700 1 Processor -- start 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.774+0000 7f09aba68700 1 -- start start 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.774+0000 7f09aba68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.774+0000 7f09aba68700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41071e0 con 0x7f09a41067f0 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a9804700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a9804700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53942/0 (socket says 172.21.15.98:53942) 2024-04-26T04:18:08.666 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a9804700 1 -- 172.21.15.98:0/2182398811 learned_addr learned my addr 172.21.15.98:0/2182398811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a9804700 1 -- 172.21.15.98:0/2182398811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a41079f0 con 0x7f09a41067f0 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a9804700 1 --2- 172.21.15.98:0/2182398811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e80 tx=0x7f099400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf76bc491ae1602 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a8802700 1 -- 172.21.15.98:0/2182398811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0994017070 con 0x7f09a41067f0 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a8802700 1 -- 172.21.15.98:0/2182398811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f099400b8c0 con 0x7f09a41067f0 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.775+0000 7f09a8802700 1 -- 172.21.15.98:0/2182398811 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0994004400 con 0x7f09a41067f0 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 -- 172.21.15.98:0/2182398811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 msgr2=0x7f09a4106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 --2- 172.21.15.98:0/2182398811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e80 tx=0x7f099400b330 comp rx=0 tx=0).stop 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 -- 172.21.15.98:0/2182398811 shutdown_connections 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 --2- 172.21.15.98:0/2182398811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a4106c10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 -- 172.21.15.98:0/2182398811 >> 172.21.15.98:0/2182398811 conn(0x7f09a4101df0 msgr2=0x7f09a4104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 -- 172.21.15.98:0/2182398811 shutdown_connections 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.776+0000 7f09aba68700 1 -- 172.21.15.98:0/2182398811 wait complete. 2024-04-26T04:18:08.667 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09aba68700 1 Processor -- start 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09aba68700 1 -- start start 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09aba68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09aba68700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09a410ac20 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a9804700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a9804700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53950/0 (socket says 172.21.15.98:53950) 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a9804700 1 -- 172.21.15.98:0/2167522893 learned_addr learned my addr 172.21.15.98:0/2167522893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a9804700 1 -- 172.21.15.98:0/2167522893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a407c080 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a9804700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0994004000 tx=0x7f0994004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0994020020 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.777+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f099400be50 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a407a270 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0994017040 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a407a7b0 con 0x7f09a41067f0 2024-04-26T04:18:08.668 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0994023020 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09a2ffd700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.778+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a4063850 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a2ffd700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a2ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a2ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a2ffd700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 0x7f098c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f099400f050 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a9003700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 0x7f098c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.779+0000 7f09a9003700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 0x7f098c03b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:07.784+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f099402a020 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.087+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f09a407ad00 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.090+0000 7f09a2ffd700 1 -- 172.21.15.98:0/2167522893 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f0994007310 con 0x7f09a41067f0 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 10 mgrc shutdown 2024-04-26T04:18:08.669 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 msgr2=0x7f098c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 0x7f098c03b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 msgr2=0x7f09a407bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0994004000 tx=0x7f0994004a30 comp rx=0 tx=0).stop 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 shutdown_connections 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f098c0390f0 0x7f098c03b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 --2- 172.21.15.98:0/2167522893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09a41067f0 0x7f09a407bb40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 >> 172.21.15.98:0/2167522893 conn(0x7f09a4101df0 msgr2=0x7f09a41028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 shutdown_connections 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-26T04:18:08.092+0000 7f09aba68700 1 -- 172.21.15.98:0/2167522893 wait complete. 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.670 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:08.671 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-04-26T04:18:08.701 INFO:tasks.cephadm:Fetching config... 2024-04-26T04:18:08.702 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:08.702 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T04:18:08.718 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T04:18:08.718 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:08.718 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T04:18:08.773 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T04:18:08.773 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:08.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T04:18:08.836 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T04:18:08.837 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:08.837 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T04:18:08.891 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T04:18:08.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1tKQV6ocTg3ya8EUdsmK+Yf+rBmfazvvZ897Eylrm5BZJW/4aZb32uUwJ2fOriRl7NmJ9NKV9Juh4Co3xudlym7phhnmgaatkZouI08KB3q7CtG0tNttnTeLrTRle47wqXD9PRDvhEO3Fgqj5UmsZ674B4ThOqIl6X7Je1hsFvcX7hcpwWNtIZaikXQC6YkeRvugfASk2YZbl/be8XJCG9mHwUXmOteb6TX8iOcagvQ3b7YeBzrML46A58dzCXeWaIt96iAyGPe3QUSV6i2pZOEbt/Y1TSQjhW17CZUgK8IvGWPlPX7GUQIXaJ5S0ib55Y7Hbl4Wc48Bq5Da8ZaT/8NZKCA/kVLqg+6rkavwfteeOfvyUuR0VWktTeB9d3GkIUGVJVJPh7vpIHuZvi9fVK6ljkzTkXi0VMxacO3K74xT5/uckoO902a+CKl4PAIfduUgN9RtJelAbnLXUa4leGVMOi71QpaLvBojeTbuPyV3XewsoEM2X2BnJ2CFiz1E= ceph-b735c86e-0383-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T04:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1tKQV6ocTg3ya8EUdsmK+Yf+rBmfazvvZ897Eylrm5BZJW/4aZb32uUwJ2fOriRl7NmJ9NKV9Juh4Co3xudlym7phhnmgaatkZouI08KB3q7CtG0tNttnTeLrTRle47wqXD9PRDvhEO3Fgqj5UmsZ674B4ThOqIl6X7Je1hsFvcX7hcpwWNtIZaikXQC6YkeRvugfASk2YZbl/be8XJCG9mHwUXmOteb6TX8iOcagvQ3b7YeBzrML46A58dzCXeWaIt96iAyGPe3QUSV6i2pZOEbt/Y1TSQjhW17CZUgK8IvGWPlPX7GUQIXaJ5S0ib55Y7Hbl4Wc48Bq5Da8ZaT/8NZKCA/kVLqg+6rkavwfteeOfvyUuR0VWktTeB9d3GkIUGVJVJPh7vpIHuZvi9fVK6ljkzTkXi0VMxacO3K74xT5/uckoO902a+CKl4PAIfduUgN9RtJelAbnLXUa4leGVMOi71QpaLvBojeTbuPyV3XewsoEM2X2BnJ2CFiz1E= ceph-b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:18:08.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1tKQV6ocTg3ya8EUdsmK+Yf+rBmfazvvZ897Eylrm5BZJW/4aZb32uUwJ2fOriRl7NmJ9NKV9Juh4Co3xudlym7phhnmgaatkZouI08KB3q7CtG0tNttnTeLrTRle47wqXD9PRDvhEO3Fgqj5UmsZ674B4ThOqIl6X7Je1hsFvcX7hcpwWNtIZaikXQC6YkeRvugfASk2YZbl/be8XJCG9mHwUXmOteb6TX8iOcagvQ3b7YeBzrML46A58dzCXeWaIt96iAyGPe3QUSV6i2pZOEbt/Y1TSQjhW17CZUgK8IvGWPlPX7GUQIXaJ5S0ib55Y7Hbl4Wc48Bq5Da8ZaT/8NZKCA/kVLqg+6rkavwfteeOfvyUuR0VWktTeB9d3GkIUGVJVJPh7vpIHuZvi9fVK6ljkzTkXi0VMxacO3K74xT5/uckoO902a+CKl4PAIfduUgN9RtJelAbnLXUa4leGVMOi71QpaLvBojeTbuPyV3XewsoEM2X2BnJ2CFiz1E= ceph-b735c86e-0383-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T04:18:09.014 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1tKQV6ocTg3ya8EUdsmK+Yf+rBmfazvvZ897Eylrm5BZJW/4aZb32uUwJ2fOriRl7NmJ9NKV9Juh4Co3xudlym7phhnmgaatkZouI08KB3q7CtG0tNttnTeLrTRle47wqXD9PRDvhEO3Fgqj5UmsZ674B4ThOqIl6X7Je1hsFvcX7hcpwWNtIZaikXQC6YkeRvugfASk2YZbl/be8XJCG9mHwUXmOteb6TX8iOcagvQ3b7YeBzrML46A58dzCXeWaIt96iAyGPe3QUSV6i2pZOEbt/Y1TSQjhW17CZUgK8IvGWPlPX7GUQIXaJ5S0ib55Y7Hbl4Wc48Bq5Da8ZaT/8NZKCA/kVLqg+6rkavwfteeOfvyUuR0VWktTeB9d3GkIUGVJVJPh7vpIHuZvi9fVK6ljkzTkXi0VMxacO3K74xT5/uckoO902a+CKl4PAIfduUgN9RtJelAbnLXUa4leGVMOi71QpaLvBojeTbuPyV3XewsoEM2X2BnJ2CFiz1E= ceph-b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:18:09.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T04:18:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:09.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:09 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2167522893' entity='client.admin' 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 -- 172.21.15.98:0/3323741284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 msgr2=0x7f760c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 --2- 172.21.15.98:0/3323741284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c1020a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75fc009910 tx=0x7f75fc009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 -- 172.21.15.98:0/3323741284 shutdown_connections 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 --2- 172.21.15.98:0/3323741284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c1020a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 -- 172.21.15.98:0/3323741284 >> 172.21.15.98:0/3323741284 conn(0x7f760c0fd220 msgr2=0x7f760c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 -- 172.21.15.98:0/3323741284 shutdown_connections 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.369+0000 7f7611344700 1 -- 172.21.15.98:0/3323741284 wait complete. 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f7611344700 1 Processor -- start 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f7611344700 1 -- start start 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f7611344700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f7611344700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f760c107210 con 0x7f760c101c80 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f760affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f760affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45696/0 (socket says 172.21.15.98:45696) 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.370+0000 7f760affd700 1 -- 172.21.15.98:0/3699018201 learned_addr learned my addr 172.21.15.98:0/3699018201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f760affd700 1 -- 172.21.15.98:0/3699018201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760c0763f0 con 0x7f760c101c80 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f760affd700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75fc000c00 tx=0x7f75fc004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75fc027070 con 0x7f760c101c80 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760c0745e0 con 0x7f760c101c80 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f75fc00f460 con 0x7f760c101c80 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75fc017440 con 0x7f760c101c80 2024-04-26T04:18:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.371+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f760c074b20 con 0x7f760c101c80 2024-04-26T04:18:11.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f75fc02a070 con 0x7f760c101c80 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f760c107990 con 0x7f760c101c80 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 0x7f75f403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f75fc04d3c0 con 0x7f760c101c80 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.372+0000 7f760a7fc700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 0x7f75f403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:11.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.373+0000 7f760a7fc700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 0x7f75f403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7600006fd0 tx=0x7f7600006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:11.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.381+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f75fc013070 con 0x7f760c101c80 2024-04-26T04:18:11.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.583+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f760c190060 con 0x7f760c101c80 2024-04-26T04:18:11.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.585+0000 7f75f3fff700 1 -- 172.21.15.98:0/3699018201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f75fc031020 con 0x7f760c101c80 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 10 mgrc shutdown 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 msgr2=0x7f75f403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 0x7f75f403b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7600006fd0 tx=0x7f7600006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 msgr2=0x7f760c075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75fc000c00 tx=0x7f75fc004f80 comp rx=0 tx=0).stop 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 shutdown_connections 2024-04-26T04:18:11.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f75f40390f0 0x7f75f403b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:11.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 --2- 172.21.15.98:0/3699018201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f760c101c80 0x7f760c075eb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:11.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 >> 172.21.15.98:0/3699018201 conn(0x7f760c0fd220 msgr2=0x7f760c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:11.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 shutdown_connections 2024-04-26T04:18:11.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:11.587+0000 7f7611344700 1 -- 172.21.15.98:0/3699018201 wait complete. 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:11.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:11 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:12.215 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T04:18:12.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T04:18:12.448 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:12.882 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:12 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3699018201' entity='client.admin' 2024-04-26T04:18:12.882 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:12.882 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 -- 172.21.15.98:0/393709743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 msgr2=0x7fcb40100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 --2- 172.21.15.98:0/393709743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb40100f90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009910 tx=0x7fcb30009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 -- 172.21.15.98:0/393709743 shutdown_connections 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 --2- 172.21.15.98:0/393709743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb40100f90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 -- 172.21.15.98:0/393709743 >> 172.21.15.98:0/393709743 conn(0x7fcb400fc130 msgr2=0x7fcb400fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 -- 172.21.15.98:0/393709743 shutdown_connections 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.023+0000 7fcb462f3700 1 -- 172.21.15.98:0/393709743 wait complete. 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb462f3700 1 Processor -- start 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb462f3700 1 -- start start 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb462f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:14.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb462f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb40106100 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45710/0 (socket says 172.21.15.98:45710) 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb3ffff700 1 -- 172.21.15.98:0/1502439878 learned_addr learned my addr 172.21.15.98:0/1502439878 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.024+0000 7fcb3ffff700 1 -- 172.21.15.98:0/1502439878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb40195e00 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb3ffff700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb30027070 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcb3000f460 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb40196090 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb30017440 con 0x7fcb40100b70 2024-04-26T04:18:14.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.025+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb40198d80 con 0x7fcb40100b70 2024-04-26T04:18:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fcb3002a070 con 0x7fcb40100b70 2024-04-26T04:18:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb40106880 con 0x7fcb40100b70 2024-04-26T04:18:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:14.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:14.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 0x7fcb2c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:14.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcb3004d3f0 con 0x7fcb40100b70 2024-04-26T04:18:14.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3f7fe700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 0x7fcb2c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:14.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.026+0000 7fcb3f7fe700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 0x7fcb2c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcb28006fd0 tx=0x7fcb28006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:14.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.031+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb30013070 con 0x7fcb40100b70 2024-04-26T04:18:14.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.235+0000 7fcb462f3700 20 mgrc start_command cmd: [{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:14.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.235+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fcb4018dd60 con 0x7fcb2c0390f0 2024-04-26T04:18:14.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.238+0000 7fcb3d7fa700 1 -- 172.21.15.98:0/1502439878 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcb4018dd60 con 0x7fcb2c0390f0 2024-04-26T04:18:14.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.238+0000 7fcb3d7fa700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 10 mgrc shutdown 2024-04-26T04:18:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 msgr2=0x7fcb2c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 0x7fcb2c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcb28006fd0 tx=0x7fcb28006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 msgr2=0x7fcb401958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:14.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30004f80 comp rx=0 tx=0).stop 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 shutdown_connections 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcb2c0390f0 0x7fcb2c03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 --2- 172.21.15.98:0/1502439878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb40100b70 0x7fcb401958c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 >> 172.21.15.98:0/1502439878 conn(0x7fcb400fc130 msgr2=0x7fcb400fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 shutdown_connections 2024-04-26T04:18:14.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:14.240+0000 7fcb462f3700 1 -- 172.21.15.98:0/1502439878 wait complete. 2024-04-26T04:18:14.974 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-04-26T04:18:14.974 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:18:14.975 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-04-26T04:18:14.993 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:18:14.993 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:15.051 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-04-26T04:18:15.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch host add smithi165 2024-04-26T04:18:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-26T04:18:15.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: Updating smithi098:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:18:15.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:15.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:15.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:15.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:15 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:16.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:16 smithi098 ceph-mon[27184]: Updating smithi098:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T04:18:16.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 -- 172.21.15.98:0/342358587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 msgr2=0x7f0d84107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 --2- 172.21.15.98:0/342358587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d84107e20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009910 tx=0x7f0d74009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 -- 172.21.15.98:0/342358587 shutdown_connections 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 --2- 172.21.15.98:0/342358587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d84107e20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 -- 172.21.15.98:0/342358587 >> 172.21.15.98:0/342358587 conn(0x7f0d840fd550 msgr2=0x7f0d840ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 -- 172.21.15.98:0/342358587 shutdown_connections 2024-04-26T04:18:16.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.714+0000 7f0d83fff700 1 -- 172.21.15.98:0/342358587 wait complete. 2024-04-26T04:18:16.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.715+0000 7f0d83fff700 1 Processor -- start 2024-04-26T04:18:16.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.715+0000 7f0d83fff700 1 -- start start 2024-04-26T04:18:16.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d83fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d83fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8410b570 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45730/0 (socket says 172.21.15.98:45730) 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d82ffd700 1 -- 172.21.15.98:0/3561268142 learned_addr learned my addr 172.21.15.98:0/3561268142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d82ffd700 1 -- 172.21.15.98:0/3561268142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8419cce0 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d82ffd700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d74006010 tx=0x7f0d74005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d74026070 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d7400b6f0 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d740037f0 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8419cfd0 con 0x7f0d84107a40 2024-04-26T04:18:16.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.716+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d84196960 con 0x7f0d84107a40 2024-04-26T04:18:16.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0d74029070 con 0x7f0d84107a40 2024-04-26T04:18:16.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:16.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d84103180 con 0x7f0d84107a40 2024-04-26T04:18:16.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 0x7f0d6c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0d7401e070 con 0x7f0d84107a40 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.717+0000 7f0d827fc700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 0x7f0d6c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:16.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.718+0000 7f0d827fc700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 0x7f0d6c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0d78006fd0 tx=0x7f0d78006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:16.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.722+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d74013070 con 0x7f0d84107a40 2024-04-26T04:18:16.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.921+0000 7f0d83fff700 20 mgrc start_command cmd: [{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:16.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:16.921+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}) -- 0x7f0d84197e90 con 0x7f0d6c039150 2024-04-26T04:18:18.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:17 smithi098 ceph-mon[27184]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:18.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:17 smithi098 ceph-mon[27184]: Deploying cephadm binary to smithi165 2024-04-26T04:18:20.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:20 smithi098 ceph-mon[27184]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:21.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.007+0000 7f0d6bfff700 1 -- 172.21.15.98:0/3561268142 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0d84197e90 con 0x7f0d6c039150 2024-04-26T04:18:21.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.007+0000 7f0d6bfff700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:21.009 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 10 mgrc shutdown 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 msgr2=0x7f0d6c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 0x7f0d6c03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0d78006fd0 tx=0x7f0d78006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 msgr2=0x7f0d8419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0d74006010 tx=0x7f0d74005030 comp rx=0 tx=0).stop 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 shutdown_connections 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0d6c039150 0x7f0d6c03b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 --2- 172.21.15.98:0/3561268142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d84107a40 0x7f0d8419c7a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 >> 172.21.15.98:0/3561268142 conn(0x7f0d840fd550 msgr2=0x7f0d84109a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 shutdown_connections 2024-04-26T04:18:21.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:21.009+0000 7f0d83fff700 1 -- 172.21.15.98:0/3561268142 wait complete. 2024-04-26T04:18:21.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T04:18:21.912 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:22.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:22 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:22.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:22 smithi098 ceph-mon[27184]: Added host smithi165 2024-04-26T04:18:22.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:22 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:22.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:22 smithi098 ceph-mon[27184]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:22.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:22 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:23.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:23 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 -- 172.21.15.98:0/4236624525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 msgr2=0x7f7e58106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 --2- 172.21.15.98:0/4236624525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58106f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7e54009910 tx=0x7f7e54009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 -- 172.21.15.98:0/4236624525 shutdown_connections 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 --2- 172.21.15.98:0/4236624525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58106f70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 -- 172.21.15.98:0/4236624525 >> 172.21.15.98:0/4236624525 conn(0x7f7e580fc610 msgr2=0x7f7e580fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 -- 172.21.15.98:0/4236624525 shutdown_connections 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.485+0000 7f7e5f472700 1 -- 172.21.15.98:0/4236624525 wait complete. 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5f472700 1 Processor -- start 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5f472700 1 -- start start 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5f472700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5f472700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5810a630 con 0x7f7e58106b90 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5d20e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5d20e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57502/0 (socket says 172.21.15.98:57502) 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5d20e700 1 -- 172.21.15.98:0/450076747 learned_addr learned my addr 172.21.15.98:0/450076747 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.486+0000 7f7e5d20e700 1 -- 172.21.15.98:0/450076747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e581972b0 con 0x7f7e58106b90 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e5d20e700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7e54006010 tx=0x7f7e54005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e54026070 con 0x7f7e58106b90 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e581975a0 con 0x7f7e58106b90 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7e5400b6f0 con 0x7f7e58106b90 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e540037f0 con 0x7f7e58106b90 2024-04-26T04:18:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.487+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5819b640 con 0x7f7e58106b90 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7e54029070 con 0x7f7e58106b90 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e58102200 con 0x7f7e58106b90 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 0x7f7e4403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7e5401e070 con 0x7f7e58106b90 2024-04-26T04:18:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e5ca0d700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 0x7f7e4403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.488+0000 7f7e5ca0d700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 0x7f7e4403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e48006fd0 tx=0x7f7e48006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.493+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e54013070 con 0x7f7e58106b90 2024-04-26T04:18:23.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.693+0000 7f7e5f472700 20 mgrc start_command cmd: [{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:18:23.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.693+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e58063850 con 0x7f7e440390f0 2024-04-26T04:18:23.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.695+0000 7f7e4e7fc700 1 -- 172.21.15.98:0/450076747 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f7e58063850 con 0x7f7e440390f0 2024-04-26T04:18:23.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.695+0000 7f7e4e7fc700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:18:23.697 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 10 mgrc shutdown 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 msgr2=0x7f7e4403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 0x7f7e4403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e48006fd0 tx=0x7f7e48006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 msgr2=0x7f7e58196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7e54006010 tx=0x7f7e54005030 comp rx=0 tx=0).stop 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 shutdown_connections 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7e440390f0 0x7f7e4403b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 --2- 172.21.15.98:0/450076747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e58106b90 0x7f7e58196d70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 >> 172.21.15.98:0/450076747 conn(0x7f7e580fc610 msgr2=0x7f7e5806a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 shutdown_connections 2024-04-26T04:18:23.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:23.697+0000 7f7e5f472700 1 -- 172.21.15.98:0/450076747 wait complete. 2024-04-26T04:18:24.311 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T04:18:24.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T04:18:24.544 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:24.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:24 smithi098 ceph-mon[27184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:24.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:24 smithi098 ceph-mon[27184]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:18:24.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:24 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:26.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 -- 172.21.15.98:0/4257249538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 msgr2=0x7fad24102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 --2- 172.21.15.98:0/4257249538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24102240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00b0a0 tx=0x7fad1c00b3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 -- 172.21.15.98:0/4257249538 shutdown_connections 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 --2- 172.21.15.98:0/4257249538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24102240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 -- 172.21.15.98:0/4257249538 >> 172.21.15.98:0/4257249538 conn(0x7fad240fd730 msgr2=0x7fad240ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 -- 172.21.15.98:0/4257249538 shutdown_connections 2024-04-26T04:18:26.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.108+0000 7fad2c400700 1 -- 172.21.15.98:0/4257249538 wait complete. 2024-04-26T04:18:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.109+0000 7fad2c400700 1 Processor -- start 2024-04-26T04:18:26.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.109+0000 7fad2c400700 1 -- start start 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2c400700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2c400700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad2410b730 con 0x7fad24101e60 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2a19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2a19c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57516/0 (socket says 172.21.15.98:57516) 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2a19c700 1 -- 172.21.15.98:0/979961012 learned_addr learned my addr 172.21.15.98:0/979961012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2a19c700 1 -- 172.21.15.98:0/979961012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad24197310 con 0x7fad24101e60 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2a19c700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad1c015040 tx=0x7fad1c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad1c017040 con 0x7fad24101e60 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fad1c022070 con 0x7fad24101e60 2024-04-26T04:18:26.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad241975a0 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fad1c004830 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.110+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2419b6a0 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fad1c027460 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad24103340 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 0x7fad1003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.111+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fad1c00f050 con 0x7fad24101e60 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.112+0000 7fad2999b700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 0x7fad1003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:26.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.112+0000 7fad2999b700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 0x7fad1003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad14006fd0 tx=0x7fad14006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.116+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fad1c0255c0 con 0x7fad24101e60 2024-04-26T04:18:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.316+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fad24063850 con 0x7fad24101e60 2024-04-26T04:18:26.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.764+0000 7fad1b7fe700 1 -- 172.21.15.98:0/979961012 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fad1c0278a0 con 0x7fad24101e60 2024-04-26T04:18:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 10 mgrc shutdown 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 msgr2=0x7fad1003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 0x7fad1003b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad14006fd0 tx=0x7fad14006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 msgr2=0x7fad24196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad1c015040 tx=0x7fad1c00bd40 comp rx=0 tx=0).stop 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 shutdown_connections 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fad100390f0 0x7fad1003b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 --2- 172.21.15.98:0/979961012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad24101e60 0x7fad24196dd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 >> 172.21.15.98:0/979961012 conn(0x7fad240fd730 msgr2=0x7fad24109b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 shutdown_connections 2024-04-26T04:18:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:26.767+0000 7fad2c400700 1 -- 172.21.15.98:0/979961012 wait complete. 2024-04-26T04:18:26.770 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-04-26T04:18:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:26 smithi098 ceph-mon[27184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:26 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/979961012' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T04:18:27.457 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-04-26T04:18:27.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch apply mon '1;smithi098:172.21.15.98=a' 2024-04-26T04:18:27.693 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:18:28.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/979961012' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T04:18:28.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: Updating smithi165:/etc/ceph/ceph.conf 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: Updating smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: Updating smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:28.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:28.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:27 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:29.117 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 -- 172.21.15.165:0/3347506339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 msgr2=0x7eff70102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 --2- 172.21.15.165:0/3347506339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff70102240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7eff60009910 tx=0x7eff60009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 -- 172.21.15.165:0/3347506339 shutdown_connections 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 --2- 172.21.15.165:0/3347506339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff70102240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 -- 172.21.15.165:0/3347506339 >> 172.21.15.165:0/3347506339 conn(0x7eff700fd730 msgr2=0x7eff700ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 -- 172.21.15.165:0/3347506339 shutdown_connections 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.115+0000 7eff7618a700 1 -- 172.21.15.165:0/3347506339 wait complete. 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff7618a700 1 Processor -- start 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff7618a700 1 -- start start 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff7618a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff7618a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff7010b730 con 0x7eff70101e60 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.165:43454/0 (socket says 172.21.15.165:43454) 2024-04-26T04:18:29.118 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.116+0000 7eff6f7fe700 1 -- 172.21.15.165:0/2687654414 learned_addr learned my addr 172.21.15.165:0/2687654414 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-26T04:18:29.119 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff6f7fe700 1 -- 172.21.15.165:0/2687654414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff700763e0 con 0x7eff70101e60 2024-04-26T04:18:29.119 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff6f7fe700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff60004f80 tx=0x7eff60005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:29.122 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff60026070 con 0x7eff70101e60 2024-04-26T04:18:29.122 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff70074a90 con 0x7eff70101e60 2024-04-26T04:18:29.122 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff6000b750 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff60003930 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.117+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff70074f50 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7eff60029070 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff70103340 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 0x7eff5803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.118+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7eff6001e070 con 0x7eff70101e60 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.119+0000 7eff6effd700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 0x7eff5803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.123 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.119+0000 7eff6effd700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 0x7eff5803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff64006fd0 tx=0x7eff64006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:29.125 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.124+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff60013070 con 0x7eff70101e60 2024-04-26T04:18:29.334 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.333+0000 7eff7618a700 20 mgrc start_command cmd: [{"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi098:172.21.15.98=a", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:29.334 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.333+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi098:172.21.15.98=a", "target": ["mon-mgr", ""]}) -- 0x7eff70063850 con 0x7eff580390f0 2024-04-26T04:18:29.340 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.339+0000 7eff6cff9700 1 -- 172.21.15.165:0/2687654414 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7eff70063850 con 0x7eff580390f0 2024-04-26T04:18:29.340 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.339+0000 7eff6cff9700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:29.340 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 10 mgrc shutdown 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 msgr2=0x7eff5803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 0x7eff5803b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff64006fd0 tx=0x7eff64006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 msgr2=0x7eff700744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7eff60004f80 tx=0x7eff60005060 comp rx=0 tx=0).stop 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 shutdown_connections 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.341+0000 7eff7618a700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7eff580390f0 0x7eff5803b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.342+0000 7eff7618a700 1 --2- 172.21.15.165:0/2687654414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff70101e60 0x7eff700744f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.342+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 >> 172.21.15.165:0/2687654414 conn(0x7eff700fd730 msgr2=0x7eff70109b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.342+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 shutdown_connections 2024-04-26T04:18:29.343 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:29.342+0000 7eff7618a700 1 -- 172.21.15.165:0/2687654414 wait complete. 2024-04-26T04:18:30.004 INFO:tasks.cephadm:Waiting for 1 mons in monmap... 2024-04-26T04:18:30.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T04:18:30.284 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi098:172.21.15.98=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: Saving service mon spec with placement smithi098:172.21.15.98=a;count:1 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:18:30.424 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:30.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:30 smithi098 ceph-mon[27184]: Reconfiguring daemon mon.a on smithi098 2024-04-26T04:18:31.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- 172.21.15.165:0/3664795950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 msgr2=0x7f8454106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 --2- 172.21.15.165:0/3664795950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454106f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f844c00b0a0 tx=0x7f844c00b3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- 172.21.15.165:0/3664795950 shutdown_connections 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 --2- 172.21.15.165:0/3664795950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454106f50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- 172.21.15.165:0/3664795950 >> 172.21.15.165:0/3664795950 conn(0x7f84540fc5f0 msgr2=0x7f84540fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- 172.21.15.165:0/3664795950 shutdown_connections 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- 172.21.15.165:0/3664795950 wait complete. 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 Processor -- start 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.653+0000 7f8459447700 1 -- start start 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8459447700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:31.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8459447700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f845406ad40 con 0x7f8454106b70 2024-04-26T04:18:31.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8452ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:31.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8452ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.165:33028/0 (socket says 172.21.15.165:33028) 2024-04-26T04:18:31.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8452ffd700 1 -- 172.21.15.165:0/669986807 learned_addr learned my addr 172.21.15.165:0/669986807 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-26T04:18:31.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8452ffd700 1 -- 172.21.15.165:0/669986807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8454197310 con 0x7f8454106b70 2024-04-26T04:18:31.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8452ffd700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f844c015040 tx=0x7f844c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:31.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c017040 con 0x7f8454106b70 2024-04-26T04:18:31.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f844c022070 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84541975a0 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.654+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f844c004830 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.655+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845419b6a0 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.655+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f844c027460 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.655+0000 7f843bfff700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84541021e0 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f843bfff700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f843bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f843bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f843bfff700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 0x7f843c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f844c00f050 con 0x7f8454106b70 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f84527fc700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 0x7f843c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:31.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.656+0000 7f84527fc700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 0x7f843c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8448006fd0 tx=0x7f8448006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:31.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.661+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f844c0255c0 con 0x7f8454106b70 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.933+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8454063850 con 0x7f8454106b70 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.934+0000 7f843bfff700 1 -- 172.21.15.165:0/669986807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f844c0278a0 con 0x7f8454106b70 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"b735c86e-0383-11ef-bc93-c7b262605968","modified":"2024-04-26T04:16:57.901441Z","created":"2024-04-26T04:16:57.901441Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T04:18:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 10 mgrc shutdown 2024-04-26T04:18:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 msgr2=0x7f843c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 0x7f843c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8448006fd0 tx=0x7f8448006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 msgr2=0x7f8454196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:31.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f844c015040 tx=0x7f844c00bd40 comp rx=0 tx=0).stop 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 shutdown_connections 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f843c0390f0 0x7f843c03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 --2- 172.21.15.165:0/669986807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8454106b70 0x7f8454196dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 >> 172.21.15.165:0/669986807 conn(0x7f84540fc5f0 msgr2=0x7f8454108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 shutdown_connections 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:31.936+0000 7f8459447700 1 -- 172.21.15.165:0/669986807 wait complete. 2024-04-26T04:18:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-04-26T04:18:32.510 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T04:18:32.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T04:18:32.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:32 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:32.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:32 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:32.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:32 smithi098 ceph-mon[27184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:32.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:32 smithi098 ceph-mon[27184]: from='client.? 172.21.15.165:0/669986807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T04:18:32.756 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 -- 172.21.15.98:0/1819856972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 msgr2=0x7fc144107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 --2- 172.21.15.98:0/1819856972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144107010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc138009910 tx=0x7fc138009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 -- 172.21.15.98:0/1819856972 shutdown_connections 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 --2- 172.21.15.98:0/1819856972 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144107010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 -- 172.21.15.98:0/1819856972 >> 172.21.15.98:0/1819856972 conn(0x7fc1440fbc50 msgr2=0x7fc1440fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 -- 172.21.15.98:0/1819856972 shutdown_connections 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.251+0000 7fc143fff700 1 -- 172.21.15.98:0/1819856972 wait complete. 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc143fff700 1 Processor -- start 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc143fff700 1 -- start start 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc143fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc143fff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc14410b8b0 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc142ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc142ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54362/0 (socket says 172.21.15.98:54362) 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc142ffd700 1 -- 172.21.15.98:0/736281370 learned_addr learned my addr 172.21.15.98:0/736281370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.252+0000 7fc142ffd700 1 -- 172.21.15.98:0/736281370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc144197340 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc142ffd700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc138004f80 tx=0x7fc138005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138026070 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1441975d0 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc13800b750 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc138003930 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.253+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc144197a30 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc138029070 con 0x7fc144106c30 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:34.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc144063850 con 0x7fc144106c30 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 0x7fc12c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc13801e070 con 0x7fc144106c30 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.254+0000 7fc1427fc700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 0x7fc12c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.255+0000 7fc1427fc700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 0x7fc12c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.259+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc138013070 con 0x7fc144106c30 2024-04-26T04:18:34.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.455+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc144198230 con 0x7fc144106c30 2024-04-26T04:18:34.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.455+0000 7fc12bfff700 1 -- 172.21.15.98:0/736281370 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+175 (secure 0 0 0) 0x7fc138030020 con 0x7fc144106c30 2024-04-26T04:18:34.458 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:18:34.458 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-04-26T04:18:34.458 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:18:34.459 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 10 mgrc shutdown 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 msgr2=0x7fc12c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 0x7fc12c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 msgr2=0x7fc144196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc138004f80 tx=0x7fc138005060 comp rx=0 tx=0).stop 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 shutdown_connections 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc12c0390f0 0x7fc12c03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 --2- 172.21.15.98:0/736281370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc144106c30 0x7fc144196e00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 >> 172.21.15.98:0/736281370 conn(0x7fc1440fbc50 msgr2=0x7fc144109b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 shutdown_connections 2024-04-26T04:18:34.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:34.457+0000 7fc143fff700 1 -- 172.21.15.98:0/736281370 wait complete. 2024-04-26T04:18:34.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:34 smithi098 ceph-mon[27184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:34.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:34 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/736281370' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:35.094 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T04:18:35.094 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:35.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T04:18:35.163 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:35.163 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:35.229 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:18:35.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T04:18:35.261 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:18:35.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:18:35.329 INFO:tasks.cephadm:Adding mgr.a on smithi098 2024-04-26T04:18:35.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch apply mgr '1;smithi098=a' 2024-04-26T04:18:35.601 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 -- 172.21.15.165:0/255673973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 msgr2=0x7f7eb8103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 --2- 172.21.15.165:0/255673973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8103780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009910 tx=0x7f7eb4009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 -- 172.21.15.165:0/255673973 shutdown_connections 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 --2- 172.21.15.165:0/255673973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8103780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 -- 172.21.15.165:0/255673973 >> 172.21.15.165:0/255673973 conn(0x7f7eb80fd240 msgr2=0x7f7eb80ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.023 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 -- 172.21.15.165:0/255673973 shutdown_connections 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.020+0000 7f7ec067b700 1 -- 172.21.15.165:0/255673973 wait complete. 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ec067b700 1 Processor -- start 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ec067b700 1 -- start start 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ec067b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ec067b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb8107300 con 0x7f7eb8103360 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ebe417700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ebe417700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.165:33044/0 (socket says 172.21.15.165:33044) 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.021+0000 7f7ebe417700 1 -- 172.21.15.165:0/2029589645 learned_addr learned my addr 172.21.15.165:0/2029589645 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7ebe417700 1 -- 172.21.15.165:0/2029589645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb80763a0 con 0x7f7eb8103360 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7ebe417700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7eb40057a0 tx=0x7f7eb4004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7eb4027070 con 0x7f7eb8103360 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb80745e0 con 0x7f7eb8103360 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7eb400bdf0 con 0x7f7eb8103360 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7eb400fd80 con 0x7f7eb8103360 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.022+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb8074ac0 con 0x7f7eb8103360 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb8107a80 con 0x7f7eb8103360 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7eb402a070 con 0x7f7eb8103360 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 0x7f7ea403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7eb404d400 con 0x7f7eb8103360 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.023+0000 7f7ebdc16700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 0x7f7ea403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.024+0000 7f7ebdc16700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 0x7f7ea403b610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.028+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7eb4013070 con 0x7f7eb8103360 2024-04-26T04:18:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:36 smithi098 ceph-mon[27184]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:37.234 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.233+0000 7f7ec067b700 20 mgrc start_command cmd: [{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:37.234 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.233+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}) -- 0x7f7eb8074d40 con 0x7f7ea4039150 2024-04-26T04:18:37.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.239+0000 7f7eaf7fe700 1 -- 172.21.15.165:0/2029589645 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7eb8074d40 con 0x7f7ea4039150 2024-04-26T04:18:37.240 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.239+0000 7f7eaf7fe700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:18:37.240 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 10 mgrc shutdown 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 msgr2=0x7f7ea403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 0x7f7ea403b610 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8006e40 comp rx=0 tx=0).stop 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 msgr2=0x7f7eb8075e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7eb40057a0 tx=0x7f7eb4004f60 comp rx=0 tx=0).stop 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 shutdown_connections 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f7ea4039150 0x7f7ea403b610 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 --2- 172.21.15.165:0/2029589645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eb8103360 0x7f7eb8075e60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 >> 172.21.15.165:0/2029589645 conn(0x7f7eb80fd240 msgr2=0x7f7eb80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 shutdown_connections 2024-04-26T04:18:37.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:18:37.241+0000 7f7ec067b700 1 -- 172.21.15.165:0/2029589645 wait complete. 2024-04-26T04:18:37.828 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T04:18:37.829 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:18:37.829 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T04:18:37.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T04:18:37.845 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-26T04:18:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T04:18:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:37.899 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-26 04:18:09.650864622 +0000 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-26 04:13:41.184688894 +0000 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-26 04:13:41.184688894 +0000 2024-04-26T04:18:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-26 04:13:41.184688894 +0000 2024-04-26T04:18:37.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T04:18:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-26T04:18:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-26T04:18:37.962 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000266973 s, 1.9 MB/s 2024-04-26T04:18:37.963 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T04:18:38.019 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-26 04:18:09.681864244 +0000 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-26 04:13:41.507684780 +0000 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-26 04:13:41.507684780 +0000 2024-04-26T04:18:38.076 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-26 04:13:41.507684780 +0000 2024-04-26T04:18:38.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T04:18:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-26T04:18:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-26T04:18:38.138 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000195428 s, 2.6 MB/s 2024-04-26T04:18:38.139 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T04:18:38.195 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-26 04:18:09.711863877 +0000 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-26 04:13:41.859680296 +0000 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-26 04:13:41.859680296 +0000 2024-04-26T04:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-26 04:13:41.859680296 +0000 2024-04-26T04:18:38.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T04:18:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-26T04:18:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-26T04:18:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000230585 s, 2.2 MB/s 2024-04-26T04:18:38.314 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T04:18:38.370 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-26T04:18:38.425 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T04:18:38.425 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.425 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T04:18:38.425 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.425 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.426 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-26 04:18:09.740863523 +0000 2024-04-26T04:18:38.426 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-26 04:13:42.192676055 +0000 2024-04-26T04:18:38.426 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-26 04:13:42.192676055 +0000 2024-04-26T04:18:38.426 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-26 04:13:42.192676055 +0000 2024-04-26T04:18:38.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: Saving service mgr spec with placement smithi098=a;count:1 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: Reconfiguring mgr.a (unknown last config time)... 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.a", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T04:18:38.486 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T04:18:38.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:38.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: Reconfiguring daemon mgr.a on smithi098 2024-04-26T04:18:38.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:38 smithi098 ceph-mon[27184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:38.488 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-26T04:18:38.488 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-26T04:18:38.488 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000167507 s, 3.1 MB/s 2024-04-26T04:18:38.489 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T04:18:38.545 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:18:38.545 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T04:18:38.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T04:18:38.564 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-04-26T04:18:38.622 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-26 04:18:25.963167063 +0000 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-26 04:13:40.084247440 +0000 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-26 04:13:40.084247440 +0000 2024-04-26T04:18:38.623 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-26 04:13:40.084247440 +0000 2024-04-26T04:18:38.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T04:18:38.690 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-26T04:18:38.690 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-26T04:18:38.690 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00022526 s, 2.3 MB/s 2024-04-26T04:18:38.692 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T04:18:38.751 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-26 04:18:25.993166624 +0000 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-26 04:13:40.435241934 +0000 2024-04-26T04:18:38.809 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-26 04:13:40.435241934 +0000 2024-04-26T04:18:38.810 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-26 04:13:40.435241934 +0000 2024-04-26T04:18:38.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T04:18:38.875 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-26T04:18:38.875 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-26T04:18:38.875 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000227873 s, 2.2 MB/s 2024-04-26T04:18:38.877 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T04:18:38.935 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-04-26T04:18:38.991 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T04:18:38.991 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:38.991 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-26 04:18:26.021166214 +0000 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-26 04:13:40.768236711 +0000 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-26 04:13:40.768236711 +0000 2024-04-26T04:18:38.992 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-26 04:13:40.768236711 +0000 2024-04-26T04:18:38.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T04:18:39.056 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-26T04:18:39.057 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-26T04:18:39.057 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00021276 s, 2.4 MB/s 2024-04-26T04:18:39.058 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T04:18:39.115 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-04-26T04:18:39.170 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T04:18:39.170 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:39.170 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T04:18:39.170 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:39.170 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T04:18:39.171 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-26 04:18:26.048165818 +0000 2024-04-26T04:18:39.171 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-26 04:13:41.094231598 +0000 2024-04-26T04:18:39.171 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-26 04:13:41.094231598 +0000 2024-04-26T04:18:39.171 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-26 04:13:41.094231598 +0000 2024-04-26T04:18:39.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T04:18:39.234 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-26T04:18:39.234 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-26T04:18:39.234 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000200644 s, 2.6 MB/s 2024-04-26T04:18:39.235 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T04:18:39.292 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-04-26T04:18:39.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T04:18:39.523 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:40.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:40.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:18:40.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:39 smithi098 ceph-mon[27184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T04:18:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T04:18:41.418 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-26T04:18:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-26T04:18:41.419 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114282 s, 918 MB/s 2024-04-26T04:18:41.419 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-26T04:18:42.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-04-26T04:18:42.274 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:18:42.680 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:42 smithi098 ceph-mon[27184]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:43.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.763+0000 7fcd327c6700 1 -- 172.21.15.98:0/2173969767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 msgr2=0x7fcd2c104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.763+0000 7fcd327c6700 1 --2- 172.21.15.98:0/2173969767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c104c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009910 tx=0x7fcd1c009c20 comp rx=0 tx=0).stop 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 -- 172.21.15.98:0/2173969767 shutdown_connections 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 --2- 172.21.15.98:0/2173969767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c104c90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 -- 172.21.15.98:0/2173969767 >> 172.21.15.98:0/2173969767 conn(0x7fcd2c0fc230 msgr2=0x7fcd2c0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 -- 172.21.15.98:0/2173969767 shutdown_connections 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 -- 172.21.15.98:0/2173969767 wait complete. 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 Processor -- start 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.764+0000 7fcd327c6700 1 -- start start 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd327c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd327c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2c1072f0 con 0x7fcd2c1028a0 2024-04-26T04:18:43.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49420/0 (socket says 172.21.15.98:49420) 2024-04-26T04:18:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd2bfff700 1 -- 172.21.15.98:0/4133128152 learned_addr learned my addr 172.21.15.98:0/4133128152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:18:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd2bfff700 1 -- 172.21.15.98:0/4133128152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2c196f90 con 0x7fcd2c1028a0 2024-04-26T04:18:43.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd2bfff700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c005950 tx=0x7fcd1c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd1c027070 con 0x7fcd2c1028a0 2024-04-26T04:18:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcd1c00f460 con 0x7fcd2c1028a0 2024-04-26T04:18:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2c197220 con 0x7fcd2c1028a0 2024-04-26T04:18:43.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd1c017440 con 0x7fcd2c1028a0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.765+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2c199f10 con 0x7fcd2c1028a0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fcd1c02a070 con 0x7fcd2c1028a0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd297fa700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 0x7fcd1403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.766+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd2c107a70 con 0x7fcd2c1028a0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.767+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcd1c04d420 con 0x7fcd2c1028a0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.767+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 0x7fcd1403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:43.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.767+0000 7fcd2b7fe700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 0x7fcd1403b5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd20006fd0 tx=0x7fcd20006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:43.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.771+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcd1c013070 con 0x7fcd2c1028a0 2024-04-26T04:18:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.974+0000 7fcd327c6700 20 mgrc start_command cmd: [{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}] 2024-04-26T04:18:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:18:43.974+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fcd2c18ef50 con 0x7fcd140390f0 2024-04-26T04:18:44.865 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:44 smithi098 ceph-mon[27184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:44.865 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:44 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T04:18:44.865 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:44 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T04:18:44.866 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:44 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:45.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:45 smithi098 ceph-mon[27184]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:18:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:46 smithi098 ceph-mon[27184]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:46 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2257205900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9"}]: dispatch 2024-04-26T04:18:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:46 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2257205900' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9"}]': finished 2024-04-26T04:18:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:46 smithi098 ceph-mon[27184]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T04:18:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:46 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:18:48.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:47 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/74787559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T04:18:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:48 smithi098 ceph-mon[27184]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:50.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:50 smithi098 ceph-mon[27184]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:52.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:52 smithi098 ceph-mon[27184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:54.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:54 smithi098 ceph-mon[27184]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:54.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T04:18:54.744 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:18:55.790 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:55 smithi098 ceph-mon[27184]: Deploying daemon osd.0 on smithi098 2024-04-26T04:18:56.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:56 smithi098 ceph-mon[27184]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:18:58.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:18:58 smithi098 ceph-mon[27184]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:19:00.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:00.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:00.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:00.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:00 smithi098 ceph-mon[27184]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:19:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.629+0000 7fcd297fa700 1 -- 172.21.15.98:0/4133128152 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcd2c18ef50 con 0x7fcd140390f0 2024-04-26T04:19:02.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.629+0000 7fcd297fa700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:19:02.630 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 10 mgrc shutdown 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 msgr2=0x7fcd1403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 0x7fcd1403b5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd20006fd0 tx=0x7fcd20006e40 comp rx=0 tx=0).stop 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 msgr2=0x7fcd2c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c005950 tx=0x7fcd1c004f80 comp rx=0 tx=0).stop 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 shutdown_connections 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcd140390f0 0x7fcd1403b5b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 --2- 172.21.15.98:0/4133128152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd2c1028a0 0x7fcd2c196a50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 >> 172.21.15.98:0/4133128152 conn(0x7fcd2c0fc230 msgr2=0x7fcd2c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 shutdown_connections 2024-04-26T04:19:02.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:02.631+0000 7fcd327c6700 1 -- 172.21.15.98:0/4133128152 wait complete. 2024-04-26T04:19:02.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:19:02.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:02.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:02.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:02.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:19:02.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:02.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:02.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:02.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:03.269 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T04:19:03.271 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_3... 2024-04-26T04:19:03.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T04:19:03.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:19:03.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:03 smithi098 ceph-mon[27184]: from='osd.0 [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T04:19:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-mon[27184]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:19:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-mon[27184]: from='osd.0 [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T04:19:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-mon[27184]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T04:19:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-mon[27184]: from='osd.0 [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-26T04:19:04.838 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:19:04.838 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0[38585]: 2024-04-26T04:19:04.759+0000 7f1c91ef2700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T04:19:04.838 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:19:04 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0[38585]: 2024-04-26T04:19:04.762+0000 7f1c8dce5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:19:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:05 smithi098 ceph-mon[27184]: from='osd.0 [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-26T04:19:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:05 smithi098 ceph-mon[27184]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T04:19:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:19:05.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:19:05.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:19:06.421 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T04:19:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T04:19:06.422 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-26T04:19:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-26T04:19:06.422 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124171 s, 844 MB/s 2024-04-26T04:19:06.422 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-26T04:19:07.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: purged_snaps scrub starts 2024-04-26T04:19:07.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: purged_snaps scrub ok 2024-04-26T04:19:07.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T04:19:07.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: osd.0 [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] boot 2024-04-26T04:19:07.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T04:19:07.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:06 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T04:19:07.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-04-26T04:19:07.890 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:19:08.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:07 smithi098 ceph-mon[27184]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T04:19:08.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: pgmap v33: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: Detected new or changed devices on smithi098 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:19:08.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 -- 172.21.15.98:0/2751964922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 msgr2=0x7f9634101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 --2- 172.21.15.98:0/2751964922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634101090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9628009910 tx=0x7f9628009c20 comp rx=0 tx=0).stop 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 -- 172.21.15.98:0/2751964922 shutdown_connections 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 --2- 172.21.15.98:0/2751964922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634101090 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 -- 172.21.15.98:0/2751964922 >> 172.21.15.98:0/2751964922 conn(0x7f96340fc230 msgr2=0x7f96340fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 -- 172.21.15.98:0/2751964922 shutdown_connections 2024-04-26T04:19:09.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 -- 172.21.15.98:0/2751964922 wait complete. 2024-04-26T04:19:09.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.321+0000 7f963ab6e700 1 Processor -- start 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963ab6e700 1 -- start start 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963ab6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963ab6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9634106200 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963890a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963890a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53222/0 (socket says 172.21.15.98:53222) 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963890a700 1 -- 172.21.15.98:0/1148513337 learned_addr learned my addr 172.21.15.98:0/1148513337 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963890a700 1 -- 172.21.15.98:0/1148513337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9634196fd0 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963890a700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f96280057a0 tx=0x7f9628004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9628027070 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f962800bdf0 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9634197260 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.322+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f962800fd80 con 0x7f9634100c70 2024-04-26T04:19:09.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.323+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9634199f50 con 0x7f9634100c70 2024-04-26T04:19:09.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.323+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f962802a070 con 0x7f9634100c70 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.323+0000 7f9631ffb700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.323+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9634106980 con 0x7f9634100c70 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9631ffb700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9631ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9631ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9631ffb700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 0x7f962403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f962804d670 con 0x7f9634100c70 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9633fff700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 0x7f962403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.324+0000 7f9633fff700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 0x7f962403b610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9620006fd0 tx=0x7f9620006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.329+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9628013070 con 0x7f9634100c70 2024-04-26T04:19:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.536+0000 7f963ab6e700 20 mgrc start_command cmd: [{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}] 2024-04-26T04:19:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:09.536+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f963418ee70 con 0x7f9624039150 2024-04-26T04:19:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:09 smithi098 ceph-mon[27184]: from='client.14199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:19:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:09 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T04:19:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:09 smithi098 ceph-mon[27184]: pgmap v34: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:09 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T04:19:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:09 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:12 smithi098 ceph-mon[27184]: pgmap v35: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:12 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1846156962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ae29254-9452-4eea-b467-2f4d92354b21"}]: dispatch 2024-04-26T04:19:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:12 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1846156962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ae29254-9452-4eea-b467-2f4d92354b21"}]': finished 2024-04-26T04:19:13.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:12 smithi098 ceph-mon[27184]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T04:19:13.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:13.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:13 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3221223768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T04:19:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:14 smithi098 ceph-mon[27184]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:16.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:16 smithi098 ceph-mon[27184]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:18.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:18 smithi098 ceph-mon[27184]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:19.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:19 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T04:19:19.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:19 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:20.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:20 smithi098 ceph-mon[27184]: Deploying daemon osd.1 on smithi098 2024-04-26T04:19:20.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:20 smithi098 ceph-mon[27184]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:22.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:22 smithi098 ceph-mon[27184]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:24.881 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:24 smithi098 ceph-mon[27184]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:26.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:25 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:26.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:25 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:26.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:25 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:26.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:25 smithi098 ceph-mon[27184]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:28.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.638+0000 7f9631ffb700 1 -- 172.21.15.98:0/1148513337 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f963418ee70 con 0x7f9624039150 2024-04-26T04:19:28.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.638+0000 7f9631ffb700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:19:28.641 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 10 mgrc shutdown 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 msgr2=0x7f962403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 0x7f962403b610 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9620006fd0 tx=0x7f9620006e40 comp rx=0 tx=0).stop 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 msgr2=0x7f9634196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f96280057a0 tx=0x7f9628004f60 comp rx=0 tx=0).stop 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 shutdown_connections 2024-04-26T04:19:28.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9624039150 0x7f962403b610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 --2- 172.21.15.98:0/1148513337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9634100c70 0x7f9634196a90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 >> 172.21.15.98:0/1148513337 conn(0x7f96340fc230 msgr2=0x7f963406a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 shutdown_connections 2024-04-26T04:19:28.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:28.641+0000 7f963ab6e700 1 -- 172.21.15.98:0/1148513337 wait complete. 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:28 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:29.290 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T04:19:29.292 INFO:tasks.cephadm:Deploying osd.2 on smithi098 with /dev/vg_nvme/lv_2... 2024-04-26T04:19:29.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T04:19:29.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:29 smithi098 ceph-mon[27184]: from='osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T04:19:30.528 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:19:30.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:30 smithi098 ceph-mon[27184]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:30.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:30 smithi098 ceph-mon[27184]: from='osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T04:19:30.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:30 smithi098 ceph-mon[27184]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T04:19:30.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:30 smithi098 ceph-mon[27184]: from='osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-26T04:19:30.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:30 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:31.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:31 smithi098 ceph-mon[27184]: from='osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-26T04:19:31.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:31 smithi098 ceph-mon[27184]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T04:19:31.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:31 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T04:19:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T04:19:32.999 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-26T04:19:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-26T04:19:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012593 s, 833 MB/s 2024-04-26T04:19:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-26T04:19:32.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:32 smithi098 ceph-mon[27184]: purged_snaps scrub starts 2024-04-26T04:19:32.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:32 smithi098 ceph-mon[27184]: purged_snaps scrub ok 2024-04-26T04:19:32.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:32 smithi098 ceph-mon[27184]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:32.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:32 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:32.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:32 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:33.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-04-26T04:19:33.728 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:19:33 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1[44907]: 2024-04-26T04:19:33.475+0000 7fa2f146f700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T04:19:33.729 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:19:33 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1[44907]: 2024-04-26T04:19:33.480+0000 7fa2e925a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:19:34.071 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:19:34.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309]' entity='osd.1' 2024-04-26T04:19:34.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:19:34.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:35.672 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:35 smithi098 ceph-mon[27184]: Detected new or changed devices on smithi098 2024-04-26T04:19:35.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:35 smithi098 ceph-mon[27184]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-26T04:19:35.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:35 smithi098 ceph-mon[27184]: osd.1 [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] boot 2024-04-26T04:19:35.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:35 smithi098 ceph-mon[27184]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T04:19:35.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:35 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T04:19:35.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 -- 172.21.15.98:0/2460616162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 msgr2=0x7f95e8104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 --2- 172.21.15.98:0/2460616162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e8104bf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009910 tx=0x7f95e4009c20 comp rx=0 tx=0).stop 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 -- 172.21.15.98:0/2460616162 shutdown_connections 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 --2- 172.21.15.98:0/2460616162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e8104bf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 -- 172.21.15.98:0/2460616162 >> 172.21.15.98:0/2460616162 conn(0x7f95e80fc230 msgr2=0x7f95e80fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 -- 172.21.15.98:0/2460616162 shutdown_connections 2024-04-26T04:19:35.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.684+0000 7f95ef719700 1 -- 172.21.15.98:0/2460616162 wait complete. 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ef719700 1 Processor -- start 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ef719700 1 -- start start 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ef719700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ef719700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95e8107290 con 0x7f95e8102800 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ed4b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ed4b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39322/0 (socket says 172.21.15.98:39322) 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ed4b5700 1 -- 172.21.15.98:0/1282217007 learned_addr learned my addr 172.21.15.98:0/1282217007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ed4b5700 1 -- 172.21.15.98:0/1282217007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e8196f30 con 0x7f95e8102800 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ed4b5700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005b40 tx=0x7f95e4004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95e4027070 con 0x7f95e8102800 2024-04-26T04:19:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f95e400f460 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e8197220 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.685+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95e40175d0 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e8199eb0 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f95e402a070 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 0x7f95d403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.686+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2199+0+0 (secure 0 0 0) 0x7f95e404d6f0 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.687+0000 7f95eccb4700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 0x7f95d403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.687+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e8107a10 con 0x7f95e8102800 2024-04-26T04:19:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.687+0000 7f95eccb4700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 0x7f95d403b5b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:35.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.692+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f95e4031070 con 0x7f95e8102800 2024-04-26T04:19:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.896+0000 7f95ef719700 20 mgrc start_command cmd: [{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}] 2024-04-26T04:19:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:35.896+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f95e818eef0 con 0x7f95d40390f0 2024-04-26T04:19:36.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:36 smithi098 ceph-mon[27184]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T04:19:36.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:36 smithi098 ceph-mon[27184]: pgmap v52: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:36.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:36 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T04:19:36.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:36 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T04:19:36.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:36 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:37.745 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:37 smithi098 ceph-mon[27184]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:19:38.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:38 smithi098 ceph-mon[27184]: pgmap v53: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:39.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:39 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3845766801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "528099cf-045c-41a0-8f9d-1730e8f25ca4"}]: dispatch 2024-04-26T04:19:39.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:39 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3845766801' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "528099cf-045c-41a0-8f9d-1730e8f25ca4"}]': finished 2024-04-26T04:19:39.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:39 smithi098 ceph-mon[27184]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T04:19:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:19:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:39 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2460308769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T04:19:40.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:40 smithi098 ceph-mon[27184]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:42.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:42 smithi098 ceph-mon[27184]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:44.681 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:44 smithi098 ceph-mon[27184]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:46.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:46 smithi098 ceph-mon[27184]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:46.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:46 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T04:19:46.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:46 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:46.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:46 smithi098 ceph-mon[27184]: Deploying daemon osd.2 on smithi098 2024-04-26T04:19:48.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:48 smithi098 ceph-mon[27184]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:50.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:50 smithi098 ceph-mon[27184]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:52.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:52 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:52.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:52 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:52.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:52 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:52.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:52 smithi098 ceph-mon[27184]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:54.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:54.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:54.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:54.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:19:54.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:19:54.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:54 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.850+0000 7f95de7fc700 1 -- 172.21.15.98:0/1282217007 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f95e818eef0 con 0x7f95d40390f0 2024-04-26T04:19:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.850+0000 7f95de7fc700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:19:54.854 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 2 on host 'smithi098' 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.852+0000 7f95ef719700 10 mgrc shutdown 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.852+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 msgr2=0x7f95d403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.852+0000 7f95ef719700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 0x7f95d403b5b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8006e40 comp rx=0 tx=0).stop 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.852+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 msgr2=0x7f95e81969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.852+0000 7f95ef719700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005b40 tx=0x7f95e4004f80 comp rx=0 tx=0).stop 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 shutdown_connections 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f95d40390f0 0x7f95d403b5b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 --2- 172.21.15.98:0/1282217007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95e8102800 0x7f95e81969f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 >> 172.21.15.98:0/1282217007 conn(0x7f95e80fc230 msgr2=0x7f95e80fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 shutdown_connections 2024-04-26T04:19:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:54.853+0000 7f95ef719700 1 -- 172.21.15.98:0/1282217007 wait complete. 2024-04-26T04:19:55.463 DEBUG:teuthology.orchestra.run.smithi098:osd.2> sudo journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T04:19:55.505 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-04-26T04:19:55.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T04:19:56.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:55 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:56.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:55 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:19:56.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:55 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:19:56.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:55 smithi098 ceph-mon[27184]: from='osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T04:19:56.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:55 smithi098 ceph-mon[27184]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:19:56.941 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:56 smithi098 ceph-mon[27184]: from='osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T04:19:56.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:56 smithi098 ceph-mon[27184]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T04:19:56.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:56 smithi098 ceph-mon[27184]: from='osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-26T04:19:56.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:56 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:19:58.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:57 smithi098 ceph-mon[27184]: from='osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-26T04:19:58.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:57 smithi098 ceph-mon[27184]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T04:19:58.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:57 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:19:58.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:57 smithi098 ceph-mon[27184]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 -- 172.21.15.98:0/1226256618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 msgr2=0x7f1478101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 --2- 172.21.15.98:0/1226256618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478101070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f146c009910 tx=0x7f146c009c20 comp rx=0 tx=0).stop 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 -- 172.21.15.98:0/1226256618 shutdown_connections 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 --2- 172.21.15.98:0/1226256618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478101070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 -- 172.21.15.98:0/1226256618 >> 172.21.15.98:0/1226256618 conn(0x7f14780fc1f0 msgr2=0x7f14780fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 -- 172.21.15.98:0/1226256618 shutdown_connections 2024-04-26T04:19:59.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.116+0000 7f147e923700 1 -- 172.21.15.98:0/1226256618 wait complete. 2024-04-26T04:19:59.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:58 smithi098 ceph-mon[27184]: purged_snaps scrub starts 2024-04-26T04:19:59.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:58 smithi098 ceph-mon[27184]: purged_snaps scrub ok 2024-04-26T04:19:59.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:58 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:19:59.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:58 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f147e923700 1 Processor -- start 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f147e923700 1 -- start start 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f147e923700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f147e923700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14781061e0 con 0x7f1478100c50 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f1477fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f1477fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33556/0 (socket says 172.21.15.98:33556) 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.117+0000 7f1477fff700 1 -- 172.21.15.98:0/1344374676 learned_addr learned my addr 172.21.15.98:0/1344374676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f1477fff700 1 -- 172.21.15.98:0/1344374676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1478195f40 con 0x7f1478100c50 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f1477fff700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f146c0057a0 tx=0x7f146c004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:59.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f146c025020 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f146c00bdf0 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f146c021de0 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14781961d0 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.118+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1478069b70 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f146c028020 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 0x7f146003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2351+0+0 (secure 0 0 0) 0x7f146c050c80 con 0x7f1478100c50 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f14777fe700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 0x7f146003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:59.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.119+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1478106960 con 0x7f1478100c50 2024-04-26T04:19:59.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.120+0000 7f14777fe700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 0x7f146003b5b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1468006fd0 tx=0x7f1468006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:59.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.124+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f146c02f020 con 0x7f1478100c50 2024-04-26T04:19:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.363+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f147806a5c0 con 0x7f1478100c50 2024-04-26T04:19:59.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.364+0000 7f14757fa700 1 -- 172.21.15.98:0/1344374676 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v17) ==== 74+0+130 (secure 0 0 0) 0x7f146c013070 con 0x7f1478100c50 2024-04-26T04:19:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:19:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 10 mgrc shutdown 2024-04-26T04:19:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 msgr2=0x7f146003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:59.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 0x7f146003b5b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1468006fd0 tx=0x7f1468006e40 comp rx=0 tx=0).stop 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 msgr2=0x7f1478195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f146c0057a0 tx=0x7f146c004f60 comp rx=0 tx=0).stop 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 shutdown_connections 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f14600390f0 0x7f146003b5b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 --2- 172.21.15.98:0/1344374676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1478100c50 0x7f1478195a00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 >> 172.21.15.98:0/1344374676 conn(0x7f14780fc1f0 msgr2=0x7f14780fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 shutdown_connections 2024-04-26T04:19:59.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:19:59.367+0000 7f147e923700 1 -- 172.21.15.98:0/1344374676 wait complete. 2024-04-26T04:19:59.965 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2[51394]: 2024-04-26T04:19:59.622+0000 7fad270ab700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T04:19:59.965 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2[51394]: 2024-04-26T04:19:59.627+0000 7fad1ee96700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:20:00.107 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1714105174,"num_in_osds":3,"osd_in_since":1714105178,"num_remapped_pgs":0} 2024-04-26T04:20:00.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: Detected new or changed devices on smithi098 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1344374676' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T04:20:00.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735]' entity='osd.2' 2024-04-26T04:20:00.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:19:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:20:01.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T04:20:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:02.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:01 smithi098 ceph-mon[27184]: osd.2 [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] boot 2024-04-26T04:20:02.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:01 smithi098 ceph-mon[27184]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T04:20:02.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T04:20:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 -- 172.21.15.98:0/1275555788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 msgr2=0x7f341c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 --2- 172.21.15.98:0/1275555788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c1021a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f341400b0a0 tx=0x7f341400b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 -- 172.21.15.98:0/1275555788 shutdown_connections 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 --2- 172.21.15.98:0/1275555788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c1021a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 -- 172.21.15.98:0/1275555788 >> 172.21.15.98:0/1275555788 conn(0x7f341c0fd320 msgr2=0x7f341c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 -- 172.21.15.98:0/1275555788 shutdown_connections 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.870+0000 7f3422b93700 1 -- 172.21.15.98:0/1275555788 wait complete. 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f3422b93700 1 Processor -- start 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f3422b93700 1 -- start start 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f3422b93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f3422b93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f341c107310 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f342092f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f342092f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33572/0 (socket says 172.21.15.98:33572) 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f342092f700 1 -- 172.21.15.98:0/4085938831 learned_addr learned my addr 172.21.15.98:0/4085938831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f342092f700 1 -- 172.21.15.98:0/4085938831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0763f0 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f342092f700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3414009700 tx=0x7f34140097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.871+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3414017040 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.872+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c0745e0 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.872+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3414022070 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.872+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34140125c0 con 0x7f341c101d80 2024-04-26T04:20:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.872+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c074b20 con 0x7f341c101d80 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f341401d440 con 0x7f341c101d80 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 20 mgrc handle_mgr_map mgrmap(e 13) 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 4 mgrc handle_mgr_map Got map version 13 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 0x7f340403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c107a90 con 0x7f341c101d80 2024-04-26T04:20:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3017+0+0 (secure 0 0 0) 0x7f341400f050 con 0x7f341c101d80 2024-04-26T04:20:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f341bfff700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 0x7f340403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.873+0000 7f341bfff700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 0x7f340403b5b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3410006fd0 tx=0x7f3410006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:02.878+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f341400bd80 con 0x7f341c101d80 2024-04-26T04:20:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:02 smithi098 ceph-mon[27184]: pgmap v69: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T04:20:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T04:20:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:02 smithi098 ceph-mon[27184]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T04:20:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T04:20:03.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.076+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f341c1900f0 con 0x7f341c101d80 2024-04-26T04:20:03.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.076+0000 7f3419ffb700 1 -- 172.21.15.98:0/4085938831 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v20) ==== 74+0+130 (secure 0 0 0) 0x7f341401ae10 con 0x7f341c101d80 2024-04-26T04:20:03.078 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 10 mgrc shutdown 2024-04-26T04:20:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 msgr2=0x7f340403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 0x7f340403b5b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3410006fd0 tx=0x7f3410006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 msgr2=0x7f341c075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:03.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3414009700 tx=0x7f34140097e0 comp rx=0 tx=0).stop 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 shutdown_connections 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f34040390f0 0x7f340403b5b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 --2- 172.21.15.98:0/4085938831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f341c101d80 0x7f341c075eb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 >> 172.21.15.98:0/4085938831 conn(0x7f341c0fd320 msgr2=0x7f341c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 shutdown_connections 2024-04-26T04:20:03.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:03.078+0000 7f3422b93700 1 -- 172.21.15.98:0/4085938831 wait complete. 2024-04-26T04:20:03.709 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"num_osds":3,"num_up_osds":3,"osd_up_since":1714105200,"num_in_osds":3,"osd_in_since":1714105178,"num_remapped_pgs":0} 2024-04-26T04:20:03.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T04:20:03.731 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T04:20:03.731 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:03.731 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:03.731 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55864]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:03.956 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:04.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:03 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T04:20:04.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:03 smithi098 ceph-mon[27184]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T04:20:04.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:03 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/4085938831' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T04:20:04.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:03 smithi098 ceph-mon[27184]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:04.042 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55920]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T04:20:04.042 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55920]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:04.042 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55920]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:04.042 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:20:03 smithi098 sudo[55920]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:04.394 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55946]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T04:20:04.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55946]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:04.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55946]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:04.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55946]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:04.725 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55965]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T04:20:04.725 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55965]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:04.725 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55965]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:04.725 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55965]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:04.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:04 smithi098 ceph-mon[27184]: mgrmap e14: a(active, since 2m) 2024-04-26T04:20:04.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:04 smithi098 ceph-mon[27184]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T04:20:04.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:20:04.984 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55971]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T04:20:04.984 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:04.984 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55971]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:04.984 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55971]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:05.237 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T04:20:05.237 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:05.237 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:05.237 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:20:04 smithi098 sudo[55974]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:05.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[55997]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T04:20:05.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[55997]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:05.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[55997]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:05.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[55997]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 -- 172.21.15.98:0/1846634978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 msgr2=0x7f105c101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 --2- 172.21.15.98:0/1846634978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c101220 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f105400b0a0 tx=0x7f105400b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 -- 172.21.15.98:0/1846634978 shutdown_connections 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 --2- 172.21.15.98:0/1846634978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c101220 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 -- 172.21.15.98:0/1846634978 >> 172.21.15.98:0/1846634978 conn(0x7f105c0fc6d0 msgr2=0x7f105c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:05.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 -- 172.21.15.98:0/1846634978 shutdown_connections 2024-04-26T04:20:05.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.574+0000 7f1060bef700 1 -- 172.21.15.98:0/1846634978 wait complete. 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f1060bef700 1 Processor -- start 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f1060bef700 1 -- start start 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f1060bef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f1060bef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f105c10a710 con 0x7f105c100e40 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f105a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f105a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33620/0 (socket says 172.21.15.98:33620) 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.575+0000 7f105a59c700 1 -- 172.21.15.98:0/1475583655 learned_addr learned my addr 172.21.15.98:0/1475583655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f105a59c700 1 -- 172.21.15.98:0/1475583655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105c0fea20 con 0x7f105c100e40 2024-04-26T04:20:05.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f105a59c700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1054015040 tx=0x7f105400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1054017040 con 0x7f105c100e40 2024-04-26T04:20:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1054022070 con 0x7f105c100e40 2024-04-26T04:20:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105c0fd0c0 con 0x7f105c100e40 2024-04-26T04:20:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1054004830 con 0x7f105c100e40 2024-04-26T04:20:05.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.576+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105c0fd580 con 0x7f105c100e40 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f105c102320 con 0x7f105c100e40 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f1054003b30 con 0x7f105c100e40 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 0x7f104403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f105400f050 con 0x7f105c100e40 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.577+0000 7f1059d9b700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 0x7f104403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:05.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.578+0000 7f1059d9b700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 0x7f104403b640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1050006fd0 tx=0x7f1050006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:05.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.582+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f105401b3f0 con 0x7f105c100e40 2024-04-26T04:20:05.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.778+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f105c0fe060 con 0x7f105c100e40 2024-04-26T04:20:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[56007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T04:20:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[56007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T04:20:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[56007]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T04:20:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 sudo[56007]: pam_unix(sudo:session): session closed for user root 2024-04-26T04:20:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.779+0000 7f104b7fe700 1 -- 172.21.15.98:0/1475583655 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+7341 (secure 0 0 0) 0x7f105401ce40 con 0x7f105c100e40 2024-04-26T04:20:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"b735c86e-0383-11ef-bc93-c7b262605968","created":"2024-04-26T04:17:00.895665+0000","modified":"2024-04-26T04:20:03.843728+0000","last_up_change":"2024-04-26T04:20:00.624370+0000","last_in_change":"2024-04-26T04:19:38.552310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:20:01.835627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6803","nonce":4264535615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6805","nonce":4264535615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6809","nonce":4264535615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6807","nonce":4264535615}]},"public_addr":"172.21.15.98:6803/4264535615","cluster_addr":"172.21.15.98:6805/4264535615","heartbeat_back_addr":"172.21.15.98:6809/4264535615","heartbeat_front_addr":"172.21.15.98:6807/4264535615","state":["exists","up"]},{"osd":1,"uuid":"6ae29254-9452-4eea-b467-2f4d92354b21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6811","nonce":727668309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6813","nonce":727668309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6817","nonce":727668309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6815","nonce":727668309}]},"public_addr":"172.21.15.98:6811/727668309","cluster_addr":"172.21.15.98:6813/727668309","heartbeat_back_addr":"172.21.15.98:6817/727668309","heartbeat_front_addr":"172.21.15.98:6815/727668309","state":["exists","up"]},{"osd":2,"uuid":"528099cf-045c-41a0-8f9d-1730e8f25ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6819","nonce":1502952735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6821","nonce":1502952735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6825","nonce":1502952735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6823","nonce":1502952735}]},"public_addr":"172.21.15.98:6819/1502952735","cluster_addr":"172.21.15.98:6821/1502952735","heartbeat_back_addr":"172.21.15.98:6825/1502952735","heartbeat_front_addr":"172.21.15.98:6823/1502952735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:03.982974+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:29.800710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:56.115364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/228615395":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6801/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/860819202":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/351312909":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/422990161":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/244208816":"2024-04-27T04:17:33.390805+0000","172.21.15.98:6801/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/3330125237":"2024-04-27T04:17:33.390805+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:20:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 10 mgrc shutdown 2024-04-26T04:20:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 msgr2=0x7f104403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:05.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 0x7f104403b640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1050006fd0 tx=0x7f1050006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 msgr2=0x7f105c0fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1054015040 tx=0x7f105400bd40 comp rx=0 tx=0).stop 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 shutdown_connections 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f1044039180 0x7f104403b640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 --2- 172.21.15.98:0/1475583655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f105c100e40 0x7f105c0fcb80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 >> 172.21.15.98:0/1475583655 conn(0x7f105c0fc6d0 msgr2=0x7f105c06a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 shutdown_connections 2024-04-26T04:20:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:05.781+0000 7f1060bef700 1 -- 172.21.15.98:0/1475583655 wait complete. 2024-04-26T04:20:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 ceph-mon[27184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T04:20:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 ceph-mon[27184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T04:20:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T04:20:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 ceph-mon[27184]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:05 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1475583655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T04:20:06.395 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T04:20:01.835627+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T04:20:06.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T04:20:06.678 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- 172.21.15.98:0/2028787588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 msgr2=0x7ff640100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 --2- 172.21.15.98:0/2028787588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640100a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009910 tx=0x7ff63c009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- 172.21.15.98:0/2028787588 shutdown_connections 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 --2- 172.21.15.98:0/2028787588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640100a60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- 172.21.15.98:0/2028787588 >> 172.21.15.98:0/2028787588 conn(0x7ff6400770d0 msgr2=0x7ff6400774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- 172.21.15.98:0/2028787588 shutdown_connections 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- 172.21.15.98:0/2028787588 wait complete. 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 Processor -- start 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.231+0000 7ff647b4b700 1 -- start start 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff647b4b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff647b4b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff640109ea0 con 0x7ff640100680 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff6458e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff6458e7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33642/0 (socket says 172.21.15.98:33642) 2024-04-26T04:20:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff6458e7700 1 -- 172.21.15.98:0/206409328 learned_addr learned my addr 172.21.15.98:0/206409328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:08.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff6458e7700 1 -- 172.21.15.98:0/206409328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff640196270 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff6458e7700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff63c004f80 tx=0x7ff63c005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff63c026070 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff640196560 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff63c00b6f0 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff63c0037f0 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.232+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64019a600 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7ff63c029070 con 0x7ff640100680 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:08.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 0x7ff62c03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7ff63c01e070 con 0x7ff640100680 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff6450e6700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 0x7ff62c03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.233+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff640101ab0 con 0x7ff640100680 2024-04-26T04:20:08.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.234+0000 7ff6450e6700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 0x7ff62c03b690 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.239+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff63c030070 con 0x7ff640100680 2024-04-26T04:20:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.434+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff640063850 con 0x7ff640100680 2024-04-26T04:20:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.435+0000 7ff636ffd700 1 -- 172.21.15.98:0/206409328 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7ff63c013070 con 0x7ff640100680 2024-04-26T04:20:08.437 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 10 mgrc shutdown 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 msgr2=0x7ff62c03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 0x7ff62c03b690 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 msgr2=0x7ff640195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff63c004f80 tx=0x7ff63c005060 comp rx=0 tx=0).stop 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 shutdown_connections 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7ff62c0391d0 0x7ff62c03b690 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 --2- 172.21.15.98:0/206409328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff640100680 0x7ff640195d30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:08.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 >> 172.21.15.98:0/206409328 conn(0x7ff6400770d0 msgr2=0x7ff640108190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:08.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 shutdown_connections 2024-04-26T04:20:08.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:08.437+0000 7ff647b4b700 1 -- 172.21.15.98:0/206409328 wait complete. 2024-04-26T04:20:08.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:08 smithi098 ceph-mon[27184]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:08.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:08 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/206409328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T04:20:09.052 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T04:20:09.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T04:20:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- 172.21.15.98:0/3689847242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 msgr2=0x7f4a9c102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 --2- 172.21.15.98:0/3689847242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c102180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800b0a0 tx=0x7f4a9800b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- 172.21.15.98:0/3689847242 shutdown_connections 2024-04-26T04:20:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 --2- 172.21.15.98:0/3689847242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c102180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:10.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- 172.21.15.98:0/3689847242 >> 172.21.15.98:0/3689847242 conn(0x7f4a9c0fd320 msgr2=0x7f4a9c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- 172.21.15.98:0/3689847242 shutdown_connections 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- 172.21.15.98:0/3689847242 wait complete. 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 Processor -- start 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- start start 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.849+0000 7f4aa3743700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c1072f0 con 0x7f4a9c101d60 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa14df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa14df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39624/0 (socket says 172.21.15.98:39624) 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa14df700 1 -- 172.21.15.98:0/1734704883 learned_addr learned my addr 172.21.15.98:0/1734704883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa14df700 1 -- 172.21.15.98:0/1734704883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c0763f0 con 0x7f4a9c101d60 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa14df700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009700 tx=0x7f4a980097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a98017040 con 0x7f4a9c101d60 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a98022070 con 0x7f4a9c101d60 2024-04-26T04:20:10.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c0745e0 con 0x7f4a9c101d60 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a980125c0 con 0x7f4a9c101d60 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.850+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c074b20 con 0x7f4a9c101d60 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f4a980128c0 con 0x7f4a9c101d60 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9c107a70 con 0x7f4a9c101d60 2024-04-26T04:20:10.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 0x7f4a8803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:10.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:10 smithi098 ceph-mon[27184]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4aa0cde700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 0x7f4a8803b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:10.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.851+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f4a9800f050 con 0x7f4a9c101d60 2024-04-26T04:20:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.852+0000 7f4aa0cde700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 0x7f4a8803b640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:10.858+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a9801be00 con 0x7f4a9c101d60 2024-04-26T04:20:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.135+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f4a9c1900d0 con 0x7f4a9c101d60 2024-04-26T04:20:11.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.136+0000 7f4a927fc700 1 -- 172.21.15.98:0/1734704883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v7) ==== 170+0+59 (secure 0 0 0) 0x7f4a98026580 con 0x7f4a9c101d60 2024-04-26T04:20:11.137 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-04-26T04:20:11.137 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB7KytmuG0hCBAA0/dDV95WypP9ZTHTsM4tww== 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 10 mgrc shutdown 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 msgr2=0x7f4a8803b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 0x7f4a8803b640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 msgr2=0x7f4a9c075eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009700 tx=0x7f4a980097e0 comp rx=0 tx=0).stop 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 shutdown_connections 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a88039180 0x7f4a8803b640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 --2- 172.21.15.98:0/1734704883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a9c101d60 0x7f4a9c075eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.138+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 >> 172.21.15.98:0/1734704883 conn(0x7f4a9c0fd320 msgr2=0x7f4a9c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.139+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 shutdown_connections 2024-04-26T04:20:11.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:11.139+0000 7f4aa3743700 1 -- 172.21.15.98:0/1734704883 wait complete. 2024-04-26T04:20:11.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:11 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1734704883' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T04:20:11.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:11 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1734704883' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T04:20:11.754 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:20:11.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T04:20:11.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T04:20:11.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T04:20:12.068 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:20:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:12 smithi098 ceph-mon[27184]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-04-26T04:20:13.467 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 -- 172.21.15.165:0/3339493094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 msgr2=0x7f8220100f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:13.467 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 --2- 172.21.15.165:0/3339493094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220100f60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f821c00b0a0 tx=0x7f821c00b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:13.467 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 -- 172.21.15.165:0/3339493094 shutdown_connections 2024-04-26T04:20:13.467 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 --2- 172.21.15.165:0/3339493094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220100f60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 -- 172.21.15.165:0/3339493094 >> 172.21.15.165:0/3339493094 conn(0x7f82200fc0e0 msgr2=0x7f82200fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 -- 172.21.15.165:0/3339493094 shutdown_connections 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.463+0000 7f8226f22700 1 -- 172.21.15.165:0/3339493094 wait complete. 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8226f22700 1 Processor -- start 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8226f22700 1 -- start start 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8226f22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8226f22700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82201060d0 con 0x7f8220100b40 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8224cbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8224cbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.165:60188/0 (socket says 172.21.15.165:60188) 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.464+0000 7f8224cbe700 1 -- 172.21.15.165:0/4135792704 learned_addr learned my addr 172.21.15.165:0/4135792704 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8224cbe700 1 -- 172.21.15.165:0/4135792704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8220196f40 con 0x7f8220100b40 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8224cbe700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f821c009700 tx=0x7f821c0097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f821c017040 con 0x7f8220100b40 2024-04-26T04:20:13.468 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82201971d0 con 0x7f8220100b40 2024-04-26T04:20:13.469 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f821c022070 con 0x7f8220100b40 2024-04-26T04:20:13.469 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f821c0125c0 con 0x7f8220100b40 2024-04-26T04:20:13.469 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.465+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8220199ec0 con 0x7f8220100b40 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f821c01d440 con 0x7f8220100b40 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8220106850 con 0x7f8220100b40 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 0x7f820803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.466+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f821c00f050 con 0x7f8220100b40 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.467+0000 7f8217fff700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 0x7f820803b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.467+0000 7f8217fff700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 0x7f820803b640 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8210006fd0 tx=0x7f8210006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:13.474 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.472+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f821c01ae10 con 0x7f8220100b40 2024-04-26T04:20:13.750 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.748+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f822002daf0 con 0x7f8220100b40 2024-04-26T04:20:13.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.750+0000 7f8215ffb700 1 -- 172.21.15.165:0/4135792704 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v8) ==== 170+0+59 (secure 0 0 0) 0x7f821c01b3f0 con 0x7f8220100b40 2024-04-26T04:20:13.752 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-04-26T04:20:13.752 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB9KytmhdPCLBAAOPAA8SrU/3VNewF7X8aQcw== 2024-04-26T04:20:13.754 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 10 mgrc shutdown 2024-04-26T04:20:13.754 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 msgr2=0x7f820803b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 0x7f820803b640 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8210006fd0 tx=0x7f8210006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 msgr2=0x7f8220196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f821c009700 tx=0x7f821c0097e0 comp rx=0 tx=0).stop 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 shutdown_connections 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f8208039180 0x7f820803b640 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 --2- 172.21.15.165:0/4135792704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8220100b40 0x7f8220196a00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 >> 172.21.15.165:0/4135792704 conn(0x7f82200fc0e0 msgr2=0x7f822006a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 shutdown_connections 2024-04-26T04:20:13.755 INFO:teuthology.orchestra.run.smithi165.stderr:2024-04-26T04:20:13.752+0000 7f8226f22700 1 -- 172.21.15.165:0/4135792704 wait complete. 2024-04-26T04:20:14.336 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:20:14.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-26T04:20:14.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-26T04:20:14.386 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T04:20:14.386 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T04:20:14.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T04:20:14.631 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:14.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:14 smithi098 ceph-mon[27184]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-04-26T04:20:14.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:14 smithi098 ceph-mon[27184]: from='client.? 172.21.15.165:0/4135792704' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T04:20:14.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:14 smithi098 ceph-mon[27184]: from='client.? 172.21.15.165:0/4135792704' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 -- 172.21.15.98:0/3030410774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 msgr2=0x7f12bc102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 --2- 172.21.15.98:0/3030410774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc102350 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b0a0 tx=0x7f12b400b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 -- 172.21.15.98:0/3030410774 shutdown_connections 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 --2- 172.21.15.98:0/3030410774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc102350 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 -- 172.21.15.98:0/3030410774 >> 172.21.15.98:0/3030410774 conn(0x7f12bc0fd800 msgr2=0x7f12bc0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 -- 172.21.15.98:0/3030410774 shutdown_connections 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.076+0000 7f12c43d8700 1 -- 172.21.15.98:0/3030410774 wait complete. 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c43d8700 1 Processor -- start 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c43d8700 1 -- start start 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c43d8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c43d8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f12bc10b840 con 0x7f12bc101f70 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c2174700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c2174700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39656/0 (socket says 172.21.15.98:39656) 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c2174700 1 -- 172.21.15.98:0/1536560639 learned_addr learned my addr 172.21.15.98:0/1536560639 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.077+0000 7f12c2174700 1 -- 172.21.15.98:0/1536560639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12bc0763e0 con 0x7f12bc101f70 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12c2174700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12b4015040 tx=0x7f12b400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4017040 con 0x7f12bc101f70 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f12b4022070 con 0x7f12bc101f70 2024-04-26T04:20:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12bc074a90 con 0x7f12bc101f70 2024-04-26T04:20:16.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4004830 con 0x7f12bc101f70 2024-04-26T04:20:16.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.078+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12bc074f50 con 0x7f12bc101f70 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f12b4003b30 con 0x7f12bc101f70 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12bc103450 con 0x7f12bc101f70 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 0x7f12a803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f12b400f050 con 0x7f12bc101f70 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12c1973700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 0x7f12a803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.079+0000 7f12c1973700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 0x7f12a803b690 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006fd0 tx=0x7f12ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:16.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.084+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12b401cbe0 con 0x7f12bc101f70 2024-04-26T04:20:16.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.335+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f12bc063850 con 0x7f12bc101f70 2024-04-26T04:20:16.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.337+0000 7f12b37fe700 1 -- 172.21.15.98:0/1536560639 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v14) ==== 74+0+88719 (secure 0 0 0) 0x7f12b401b3f0 con 0x7f12bc101f70 2024-04-26T04:20:16.339 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:16.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.339+0000 7f12c43d8700 10 mgrc shutdown 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.339+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 msgr2=0x7f12a803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.339+0000 7f12c43d8700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 0x7f12a803b690 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006fd0 tx=0x7f12ac006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.339+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 msgr2=0x7f12bc0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.339+0000 7f12c43d8700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12b4015040 tx=0x7f12b400bd40 comp rx=0 tx=0).stop 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 shutdown_connections 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f12a80391d0 0x7f12a803b690 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 --2- 172.21.15.98:0/1536560639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12bc101f70 0x7f12bc0744f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 >> 172.21.15.98:0/1536560639 conn(0x7f12bc0fd800 msgr2=0x7f12bc109c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 shutdown_connections 2024-04-26T04:20:16.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:16.340+0000 7f12c43d8700 1 -- 172.21.15.98:0/1536560639 wait complete. 2024-04-26T04:20:16.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:16 smithi098 ceph-mon[27184]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-26T04:20:16.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:16 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1536560639' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T04:20:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":14,"flags":0,"active_gid":14150,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":2693888440},{"type":"v1","addr":"172.21.15.98:6801","nonce":2693888440}]},"active_addr":"172.21.15.98:6801/2693888440","active_change":"2024-04-26T04:17:59.527064+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1239585855}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1014680848}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3255665700}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":143022130}]}]} 2024-04-26T04:20:16.929 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T04:20:16.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T04:20:16.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T04:20:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.775+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/1717298098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 msgr2=0x7fbfe4102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.775+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/1717298098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4102350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009910 tx=0x7fbfd4009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.775+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/1717298098 shutdown_connections 2024-04-26T04:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.775+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/1717298098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4102350 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/1717298098 >> 172.21.15.98:0/1717298098 conn(0x7fbfe40fd800 msgr2=0x7fbfe40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/1717298098 shutdown_connections 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/1717298098 wait complete. 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 Processor -- start 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 -- start start 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.776+0000 7fbfeaa4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe410b840 con 0x7fbfe4101f70 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39984/0 (socket says 172.21.15.98:39984) 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe3fff700 1 -- 172.21.15.98:0/2288644702 learned_addr learned my addr 172.21.15.98:0/2288644702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe3fff700 1 -- 172.21.15.98:0/2288644702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe4197420 con 0x7fbfe4101f70 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe3fff700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4004f80 tx=0x7fbfd4005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfd4026070 con 0x7fbfe4101f70 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbfd400b6f0 con 0x7fbfe4101f70 2024-04-26T04:20:18.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe4197710 con 0x7fbfe4101f70 2024-04-26T04:20:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfd40037f0 con 0x7fbfe4101f70 2024-04-26T04:20:18.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.777+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe419b7b0 con 0x7fbfe4101f70 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fbfd4029070 con 0x7fbfe4101f70 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe4103450 con 0x7fbfe4101f70 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 0x7fbfcc03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fbfd401e070 con 0x7fbfe4101f70 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.778+0000 7fbfe37fe700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 0x7fbfcc03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:18.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.779+0000 7fbfe37fe700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 0x7fbfcc03b690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8006fd0 tx=0x7fbfd8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:18.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.783+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbfd4013070 con 0x7fbfe4101f70 2024-04-26T04:20:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.978+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfe4063850 con 0x7fbfe4101f70 2024-04-26T04:20:18.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:18 smithi098 ceph-mon[27184]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-26T04:20:18.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.979+0000 7fbfe17fa700 1 -- 172.21.15.98:0/2288644702 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+7341 (secure 0 0 0) 0x7fbfd4031cc0 con 0x7fbfe4101f70 2024-04-26T04:20:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:18.980 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"b735c86e-0383-11ef-bc93-c7b262605968","created":"2024-04-26T04:17:00.895665+0000","modified":"2024-04-26T04:20:03.843728+0000","last_up_change":"2024-04-26T04:20:00.624370+0000","last_in_change":"2024-04-26T04:19:38.552310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:20:01.835627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6803","nonce":4264535615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6805","nonce":4264535615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6809","nonce":4264535615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6807","nonce":4264535615}]},"public_addr":"172.21.15.98:6803/4264535615","cluster_addr":"172.21.15.98:6805/4264535615","heartbeat_back_addr":"172.21.15.98:6809/4264535615","heartbeat_front_addr":"172.21.15.98:6807/4264535615","state":["exists","up"]},{"osd":1,"uuid":"6ae29254-9452-4eea-b467-2f4d92354b21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6811","nonce":727668309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6813","nonce":727668309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6817","nonce":727668309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6815","nonce":727668309}]},"public_addr":"172.21.15.98:6811/727668309","cluster_addr":"172.21.15.98:6813/727668309","heartbeat_back_addr":"172.21.15.98:6817/727668309","heartbeat_front_addr":"172.21.15.98:6815/727668309","state":["exists","up"]},{"osd":2,"uuid":"528099cf-045c-41a0-8f9d-1730e8f25ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6819","nonce":1502952735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6821","nonce":1502952735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6825","nonce":1502952735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6823","nonce":1502952735}]},"public_addr":"172.21.15.98:6819/1502952735","cluster_addr":"172.21.15.98:6821/1502952735","heartbeat_back_addr":"172.21.15.98:6825/1502952735","heartbeat_front_addr":"172.21.15.98:6823/1502952735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:03.982974+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:29.800710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:56.115364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/228615395":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6801/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/860819202":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/351312909":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/422990161":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/244208816":"2024-04-27T04:17:33.390805+0000","172.21.15.98:6801/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/3330125237":"2024-04-27T04:17:33.390805+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 10 mgrc shutdown 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 msgr2=0x7fbfcc03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 0x7fbfcc03b690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8006fd0 tx=0x7fbfd8006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 msgr2=0x7fbfe4196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4004f80 tx=0x7fbfd4005060 comp rx=0 tx=0).stop 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 shutdown_connections 2024-04-26T04:20:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbfcc0391d0 0x7fbfcc03b690 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 --2- 172.21.15.98:0/2288644702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfe4101f70 0x7fbfe4196ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 >> 172.21.15.98:0/2288644702 conn(0x7fbfe40fd800 msgr2=0x7fbfe4109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 shutdown_connections 2024-04-26T04:20:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:18.981+0000 7fbfeaa4f700 1 -- 172.21.15.98:0/2288644702 wait complete. 2024-04-26T04:20:19.618 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T04:20:19.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T04:20:19.855 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:19 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2288644702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T04:20:20.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:20 smithi098 ceph-mon[27184]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-26T04:20:21.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/69621017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 msgr2=0x7fc3c4102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:21.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/69621017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4102320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009910 tx=0x7fc3b4009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:21.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/69621017 shutdown_connections 2024-04-26T04:20:21.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/69621017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4102320 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/69621017 >> 172.21.15.98:0/69621017 conn(0x7fc3c40fd7f0 msgr2=0x7fc3c40ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/69621017 shutdown_connections 2024-04-26T04:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.357+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/69621017 wait complete. 2024-04-26T04:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c9ffd700 1 Processor -- start 2024-04-26T04:20:21.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c9ffd700 1 -- start start 2024-04-26T04:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c9ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:21.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c9ffd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c410b810 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40002/0 (socket says 172.21.15.98:40002) 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c37fe700 1 -- 172.21.15.98:0/3924271244 learned_addr learned my addr 172.21.15.98:0/3924271244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c37fe700 1 -- 172.21.15.98:0/3924271244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c4197350 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.358+0000 7fc3c37fe700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4004f80 tx=0x7fc3b4005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3b4026070 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c4197640 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc3b400b6f0 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3b40037f0 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c419b6e0 con 0x7fc3c4101f40 2024-04-26T04:20:21.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fc3b4029070 con 0x7fc3c4101f40 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c17fa700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.359+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c4103420 con 0x7fc3c4101f40 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3c17fa700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3c17fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3c17fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3c17fa700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 0x7fc3a403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fc3b401e070 con 0x7fc3c4101f40 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3baffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 0x7fc3a403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:21.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.360+0000 7fc3baffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 0x7fc3a403b640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac006fd0 tx=0x7fc3ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:21.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.365+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc3b4013070 con 0x7fc3c4101f40 2024-04-26T04:20:21.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.560+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3c4198170 con 0x7fc3c4101f40 2024-04-26T04:20:21.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.561+0000 7fc3c17fa700 1 -- 172.21.15.98:0/3924271244 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+7341 (secure 0 0 0) 0x7fc3b4030020 con 0x7fc3c4101f40 2024-04-26T04:20:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:21.562 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"b735c86e-0383-11ef-bc93-c7b262605968","created":"2024-04-26T04:17:00.895665+0000","modified":"2024-04-26T04:20:03.843728+0000","last_up_change":"2024-04-26T04:20:00.624370+0000","last_in_change":"2024-04-26T04:19:38.552310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:20:01.835627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6803","nonce":4264535615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6805","nonce":4264535615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6809","nonce":4264535615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4264535615},{"type":"v1","addr":"172.21.15.98:6807","nonce":4264535615}]},"public_addr":"172.21.15.98:6803/4264535615","cluster_addr":"172.21.15.98:6805/4264535615","heartbeat_back_addr":"172.21.15.98:6809/4264535615","heartbeat_front_addr":"172.21.15.98:6807/4264535615","state":["exists","up"]},{"osd":1,"uuid":"6ae29254-9452-4eea-b467-2f4d92354b21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6811","nonce":727668309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6813","nonce":727668309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6817","nonce":727668309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":727668309},{"type":"v1","addr":"172.21.15.98:6815","nonce":727668309}]},"public_addr":"172.21.15.98:6811/727668309","cluster_addr":"172.21.15.98:6813/727668309","heartbeat_back_addr":"172.21.15.98:6817/727668309","heartbeat_front_addr":"172.21.15.98:6815/727668309","state":["exists","up"]},{"osd":2,"uuid":"528099cf-045c-41a0-8f9d-1730e8f25ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6819","nonce":1502952735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6821","nonce":1502952735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6825","nonce":1502952735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1502952735},{"type":"v1","addr":"172.21.15.98:6823","nonce":1502952735}]},"public_addr":"172.21.15.98:6819/1502952735","cluster_addr":"172.21.15.98:6821/1502952735","heartbeat_back_addr":"172.21.15.98:6825/1502952735","heartbeat_front_addr":"172.21.15.98:6823/1502952735","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:03.982974+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:29.800710+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:19:56.115364+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/228615395":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6801/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/142088859":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/860819202":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/351312909":"2024-04-27T04:17:59.526912+0000","172.21.15.98:0/422990161":"2024-04-27T04:17:59.526912+0000","172.21.15.98:6800/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/244208816":"2024-04-27T04:17:33.390805+0000","172.21.15.98:6801/2639234633":"2024-04-27T04:17:33.390805+0000","172.21.15.98:0/3330125237":"2024-04-27T04:17:33.390805+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 10 mgrc shutdown 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 msgr2=0x7fc3a403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 0x7fc3a403b640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac006fd0 tx=0x7fc3ac006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 msgr2=0x7fc3c4196e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4004f80 tx=0x7fc3b4005060 comp rx=0 tx=0).stop 2024-04-26T04:20:21.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 shutdown_connections 2024-04-26T04:20:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fc3a4039180 0x7fc3a403b640 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 --2- 172.21.15.98:0/3924271244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c4101f40 0x7fc3c4196e10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 >> 172.21.15.98:0/3924271244 conn(0x7fc3c40fd7f0 msgr2=0x7fc3c4108af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 shutdown_connections 2024-04-26T04:20:21.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:21.563+0000 7fc3c9ffd700 1 -- 172.21.15.98:0/3924271244 wait complete. 2024-04-26T04:20:21.861 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:21 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/3924271244' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T04:20:22.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T04:20:22.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T04:20:22.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T04:20:22.400 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:22.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:22 smithi098 ceph-mon[27184]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-04-26T04:20:24.899 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:24 smithi098 ceph-mon[27184]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/1960907338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 msgr2=0x7f5d08100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/1960907338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d08100a00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009910 tx=0x7f5cf8009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/1960907338 shutdown_connections 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/1960907338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d08100a00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/1960907338 >> 172.21.15.98:0/1960907338 conn(0x7f5d08077070 msgr2=0x7f5d08077480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/1960907338 shutdown_connections 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.472+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/1960907338 wait complete. 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0c9bd700 1 Processor -- start 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0c9bd700 1 -- start start 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0c9bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0c9bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810b820 con 0x7f5d08100620 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:25.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40016/0 (socket says 172.21.15.98:40016) 2024-04-26T04:20:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.473+0000 7f5d0659c700 1 -- 172.21.15.98:0/3510830113 learned_addr learned my addr 172.21.15.98:0/3510830113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5d0659c700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d08103490 con 0x7f5d08100620 2024-04-26T04:20:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5d0659c700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8004f80 tx=0x7f5cf8005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cf8026070 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5cf800b750 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5cf8003930 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d08101b40 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.474+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d08102000 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d080698c0 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f5cf8029070 con 0x7f5d08100620 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 0x7f5cf003b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f5cf801e070 con 0x7f5d08100620 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 0x7f5cf0040450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cf800b0b0 con 0x7f5cf003e010 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5d05d9b700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 0x7f5cf003b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5cf8054050 con 0x7f5d08100620 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.475+0000 7f5d06d9d700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 0x7f5cf0040450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.476+0000 7f5d05d9b700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 0x7f5cf003b690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc006fd0 tx=0x7f5cfc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.476+0000 7f5d06d9d700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 0x7f5cf0040450 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.476+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== osd.0 v2:172.21.15.98:6802/4264535615 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cf800b0b0 con 0x7f5cf003e010 2024-04-26T04:20:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.492+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 --> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d08062880 con 0x7f5cf003e010 2024-04-26T04:20:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5cf77fe700 1 -- 172.21.15.98:0/3510830113 <== osd.0 v2:172.21.15.98:6802/4264535615 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5d08062880 con 0x7f5cf003e010 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 msgr2=0x7f5cf0040450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 0x7f5cf0040450 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 10 mgrc shutdown 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 msgr2=0x7f5cf003b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 0x7f5cf003b690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc006fd0 tx=0x7f5cfc006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 msgr2=0x7f5d081015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8004f80 tx=0x7f5cf8005060 comp rx=0 tx=0).stop 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 shutdown_connections 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6802/4264535615,v1:172.21.15.98:6803/4264535615] conn(0x7f5cf003e010 0x7f5cf0040450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d08100620 0x7f5d081015a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 --2- 172.21.15.98:0/3510830113 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f5cf00391d0 0x7f5cf003b690 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 >> 172.21.15.98:0/3510830113 conn(0x7f5d08077070 msgr2=0x7f5d080ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 shutdown_connections 2024-04-26T04:20:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:25.493+0000 7f5d0c9bd700 1 -- 172.21.15.98:0/3510830113 wait complete. 2024-04-26T04:20:26.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 -- 172.21.15.98:0/29594416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 msgr2=0x7f3214101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:26.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 --2- 172.21.15.98:0/29594416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214101240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3204009910 tx=0x7f3204009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 -- 172.21.15.98:0/29594416 shutdown_connections 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 --2- 172.21.15.98:0/29594416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214101240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 -- 172.21.15.98:0/29594416 >> 172.21.15.98:0/29594416 conn(0x7f32140fc6f0 msgr2=0x7f32140feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 -- 172.21.15.98:0/29594416 shutdown_connections 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.234+0000 7f321c2b5700 1 -- 172.21.15.98:0/29594416 wait complete. 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321c2b5700 1 Processor -- start 2024-04-26T04:20:26.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321c2b5700 1 -- start start 2024-04-26T04:20:26.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321c2b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:26.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321c2b5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f321410a730 con 0x7f3214100e60 2024-04-26T04:20:26.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321a051700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321a051700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40028/0 (socket says 172.21.15.98:40028) 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321a051700 1 -- 172.21.15.98:0/1033473337 learned_addr learned my addr 172.21.15.98:0/1033473337 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.235+0000 7f321a051700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3214198210 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f321a051700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3204004f80 tx=0x7f3204005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204026070 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3214198500 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f320400b6f0 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32040037f0 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32141989c0 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.236+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3214062880 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f3204029070 con 0x7f3214100e60 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 0x7f320003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f320401e070 con 0x7f3214100e60 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f3219850700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 0x7f320003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 0x7f3200040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f320400b0b0 con 0x7f320003dfc0 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3204054050 con 0x7f3214100e60 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f321a852700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 0x7f3200040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f3219850700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 0x7f320003b640 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3210006fd0 tx=0x7f3210006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.237+0000 7f321a852700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 0x7f3200040400 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.238+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== osd.2 v2:172.21.15.98:6818/1502952735 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f320400b0b0 con 0x7f320003dfc0 2024-04-26T04:20:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 --> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3214062d10 con 0x7f320003dfc0 2024-04-26T04:20:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f320b7fe700 1 -- 172.21.15.98:0/1033473337 <== osd.2 v2:172.21.15.98:6818/1502952735 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3214062d10 con 0x7f320003dfc0 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 msgr2=0x7f3200040400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 0x7f3200040400 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 10 mgrc shutdown 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 msgr2=0x7f320003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 0x7f320003b640 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3210006fd0 tx=0x7f3210006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 msgr2=0x7f3214197cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3204004f80 tx=0x7f3204005060 comp rx=0 tx=0).stop 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 shutdown_connections 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6818/1502952735,v1:172.21.15.98:6819/1502952735] conn(0x7f320003dfc0 0x7f3200040400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3214100e60 0x7f3214197cd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 --2- 172.21.15.98:0/1033473337 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f3200039180 0x7f320003b640 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.255+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 >> 172.21.15.98:0/1033473337 conn(0x7f32140fc6f0 msgr2=0x7f3214108aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.256+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 shutdown_connections 2024-04-26T04:20:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:26.256+0000 7f321c2b5700 1 -- 172.21.15.98:0/1033473337 wait complete. 2024-04-26T04:20:26.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:26 smithi098 ceph-mon[27184]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:27.329 INFO:teuthology.orchestra.run.smithi098.stdout:34359738387 2024-04-26T04:20:27.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T04:20:27.699 INFO:teuthology.orchestra.run.smithi098.stdout:77309411339 2024-04-26T04:20:27.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T04:20:27.746 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 -- 172.21.15.98:0/1037141269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 msgr2=0x7f68f8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 --2- 172.21.15.98:0/1037141269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8100a40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009910 tx=0x7f68f4009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 -- 172.21.15.98:0/1037141269 shutdown_connections 2024-04-26T04:20:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 --2- 172.21.15.98:0/1037141269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8100a40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:28.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 -- 172.21.15.98:0/1037141269 >> 172.21.15.98:0/1037141269 conn(0x7f68f8077010 msgr2=0x7f68f8077420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 -- 172.21.15.98:0/1037141269 shutdown_connections 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.193+0000 7f68fff17700 1 -- 172.21.15.98:0/1037141269 wait complete. 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.194+0000 7f68fff17700 1 Processor -- start 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.194+0000 7f68fff17700 1 -- start start 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.194+0000 7f68fff17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.194+0000 7f68fff17700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68f810b860 con 0x7f68f8100660 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fdcb3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fdcb3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40046/0 (socket says 172.21.15.98:40046) 2024-04-26T04:20:28.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fdcb3700 1 -- 172.21.15.98:0/4228191948 learned_addr learned my addr 172.21.15.98:0/4228191948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:28.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fdcb3700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f8197390 con 0x7f68f8100660 2024-04-26T04:20:28.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fdcb3700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f68f4004f80 tx=0x7f68f4005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:28.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68f4026070 con 0x7f68f8100660 2024-04-26T04:20:28.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f68f400b6f0 con 0x7f68f8100660 2024-04-26T04:20:28.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f8197680 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68f40037f0 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.195+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f819b720 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f68f819bbf0 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f68f4029070 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68eeffd700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68eeffd700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68eeffd700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.196+0000 7f68eeffd700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68eeffd700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 0x7f68e403b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f68f401e070 con 0x7f68f8100660 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68fd4b2700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 0x7f68e403b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68eeffd700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 0x7f68e4040450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68f4004830 con 0x7f68e403e010 2024-04-26T04:20:28.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f68f4054050 con 0x7f68f8100660 2024-04-26T04:20:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68fe4b4700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 0x7f68e4040450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68fd4b2700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 0x7f68e403b690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.197+0000 7f68fe4b4700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 0x7f68e4040450 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.198+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== osd.1 v2:172.21.15.98:6810/727668309 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f68f4004830 con 0x7f68e403e010 2024-04-26T04:20:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.215+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 --> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f68f819c080 con 0x7f68e403e010 2024-04-26T04:20:28.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68eeffd700 1 -- 172.21.15.98:0/4228191948 <== osd.1 v2:172.21.15.98:6810/727668309 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f68f819c080 con 0x7f68e403e010 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 msgr2=0x7f68e4040450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 0x7f68e4040450 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 10 mgrc shutdown 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 msgr2=0x7f68e403b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 0x7f68e403b690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 msgr2=0x7f68f8196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f68f4004f80 tx=0x7f68f4005060 comp rx=0 tx=0).stop 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 shutdown_connections 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6810/727668309,v1:172.21.15.98:6811/727668309] conn(0x7f68e403e010 0x7f68e4040450 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68f8100660 0x7f68f8196e50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:28.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 --2- 172.21.15.98:0/4228191948 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f68e40391d0 0x7f68e403b690 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.216+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 >> 172.21.15.98:0/4228191948 conn(0x7f68f8077010 msgr2=0x7f68f80ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.217+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 shutdown_connections 2024-04-26T04:20:28.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:28.217+0000 7f68fff17700 1 -- 172.21.15.98:0/4228191948 wait complete. 2024-04-26T04:20:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:28 smithi098 ceph-mon[27184]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:29.432 INFO:teuthology.orchestra.run.smithi098.stdout:55834574864 2024-04-26T04:20:29.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T04:20:29.853 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:30.893 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:30 smithi098 ceph-mon[27184]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/3724138650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 msgr2=0x7fbd54107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/3724138650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54107200 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00b0a0 tx=0x7fbd4c00b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/3724138650 shutdown_connections 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/3724138650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54107200 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/3724138650 >> 172.21.15.98:0/3724138650 conn(0x7fbd540fba20 msgr2=0x7fbd540fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:32.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.100+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/3724138650 shutdown_connections 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.101+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/3724138650 wait complete. 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.101+0000 7fbd5c7e2700 1 Processor -- start 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.101+0000 7fbd5c7e2700 1 -- start start 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.101+0000 7fbd5c7e2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.101+0000 7fbd5c7e2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5410b880 con 0x7fbd54106e20 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5a57e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5a57e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32854/0 (socket says 172.21.15.98:32854) 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5a57e700 1 -- 172.21.15.98:0/45130308 learned_addr learned my addr 172.21.15.98:0/45130308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5a57e700 1 -- 172.21.15.98:0/45130308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd541973b0 con 0x7fbd54106e20 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5a57e700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0093e0 tx=0x7fbd4c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd4c017040 con 0x7fbd54106e20 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd4c022070 con 0x7fbd54106e20 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd54197640 con 0x7fbd54106e20 2024-04-26T04:20:32.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd4c004860 con 0x7fbd54106e20 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.102+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5419b740 con 0x7fbd54106e20 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fbd4c027460 con 0x7fbd54106e20 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd54101220 con 0x7fbd54106e20 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 0x7fbd4003b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fbd4c00f050 con 0x7fbd54106e20 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.103+0000 7fbd59d7d700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 0x7fbd4003b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.104+0000 7fbd59d7d700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 0x7fbd4003b6d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd44006fd0 tx=0x7fbd44006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:32.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.108+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd4c003d80 con 0x7fbd54106e20 2024-04-26T04:20:32.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.350+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbd54063850 con 0x7fbd54106e20 2024-04-26T04:20:32.351 INFO:teuthology.orchestra.run.smithi098.stdout:34359738388 2024-04-26T04:20:32.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.350+0000 7fbd4b7fe700 1 -- 172.21.15.98:0/45130308 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd4c04de20 con 0x7fbd54106e20 2024-04-26T04:20:32.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 10 mgrc shutdown 2024-04-26T04:20:32.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 msgr2=0x7fbd4003b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 0x7fbd4003b6d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd44006fd0 tx=0x7fbd44006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:32.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 msgr2=0x7fbd54196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0093e0 tx=0x7fbd4c00bd40 comp rx=0 tx=0).stop 2024-04-26T04:20:32.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 shutdown_connections 2024-04-26T04:20:32.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fbd40039210 0x7fbd4003b6d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 --2- 172.21.15.98:0/45130308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd54106e20 0x7fbd54196e70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 >> 172.21.15.98:0/45130308 conn(0x7fbd540fba20 msgr2=0x7fbd5410a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:32.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 shutdown_connections 2024-04-26T04:20:32.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.352+0000 7fbd5c7e2700 1 -- 172.21.15.98:0/45130308 wait complete. 2024-04-26T04:20:32.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 -- 172.21.15.98:0/251084199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 msgr2=0x7f15b8101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/251084199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8101b10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f15a800b0a0 tx=0x7f15a800b3b0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 -- 172.21.15.98:0/251084199 shutdown_connections 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/251084199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8101b10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 -- 172.21.15.98:0/251084199 >> 172.21.15.98:0/251084199 conn(0x7f15b8077070 msgr2=0x7f15b8077480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 -- 172.21.15.98:0/251084199 shutdown_connections 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.529+0000 7f15bc9fc700 1 -- 172.21.15.98:0/251084199 wait complete. 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15bc9fc700 1 Processor -- start 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15bc9fc700 1 -- start start 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15bc9fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:32.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15bc9fc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810a730 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32872/0 (socket says 172.21.15.98:32872) 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15b659c700 1 -- 172.21.15.98:0/2755283674 learned_addr learned my addr 172.21.15.98:0/2755283674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15b659c700 1 -- 172.21.15.98:0/2755283674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b8198260 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15b659c700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f15a80093e0 tx=0x7f15a800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.530+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a8017040 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f15a8022070 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b81984f0 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a8004890 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81989b0 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f15a8003b30 con 0x7f15b8101730 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15a77fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15a77fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:32.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15a77fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15a77fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15a77fe700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 0x7f15a003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.531+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b81024e0 con 0x7f15b8101730 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f15a800f050 con 0x7f15b8101730 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15b5d9b700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 0x7f15a003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.532+0000 7f15b5d9b700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 0x7f15a003b640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f15ac006fd0 tx=0x7f15ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:32.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.537+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15a801ce80 con 0x7f15b8101730 2024-04-26T04:20:32.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.735+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f15b8063850 con 0x7f15b8101730 2024-04-26T04:20:32.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:32 smithi098 ceph-mon[27184]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:32.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:32 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/45130308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T04:20:32.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.735+0000 7f15a77fe700 1 -- 172.21.15.98:0/2755283674 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f15a80041d0 con 0x7f15b8101730 2024-04-26T04:20:32.737 INFO:teuthology.orchestra.run.smithi098.stdout:77309411340 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 10 mgrc shutdown 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 msgr2=0x7f15a003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 0x7f15a003b640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f15ac006fd0 tx=0x7f15ac006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 msgr2=0x7f15b8197d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f15a80093e0 tx=0x7f15a800bd40 comp rx=0 tx=0).stop 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 shutdown_connections 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f15a0039180 0x7f15a003b640 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 --2- 172.21.15.98:0/2755283674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15b8101730 0x7f15b8197d20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 >> 172.21.15.98:0/2755283674 conn(0x7f15b8077070 msgr2=0x7f15b80ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 shutdown_connections 2024-04-26T04:20:32.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:32.738+0000 7f15bc9fc700 1 -- 172.21.15.98:0/2755283674 wait complete. 2024-04-26T04:20:33.310 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411340 for osd.2 2024-04-26T04:20:33.310 DEBUG:teuthology.parallel:result is None 2024-04-26T04:20:33.888 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:33 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2755283674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T04:20:34.090 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738387 got 34359738388 for osd.0 2024-04-26T04:20:34.090 DEBUG:teuthology.parallel:result is None 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 -- 172.21.15.98:0/2709074884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 msgr2=0x7fbaa8101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 --2- 172.21.15.98:0/2709074884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8101240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009910 tx=0x7fbaa4009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 -- 172.21.15.98:0/2709074884 shutdown_connections 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 --2- 172.21.15.98:0/2709074884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8101240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 -- 172.21.15.98:0/2709074884 >> 172.21.15.98:0/2709074884 conn(0x7fbaa80fc730 msgr2=0x7fbaa80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 -- 172.21.15.98:0/2709074884 shutdown_connections 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.517+0000 7fbab0596700 1 -- 172.21.15.98:0/2709074884 wait complete. 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbab0596700 1 Processor -- start 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbab0596700 1 -- start start 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbab0596700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbab0596700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa810a730 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbaae332700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbaae332700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32896/0 (socket says 172.21.15.98:32896) 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbaae332700 1 -- 172.21.15.98:0/4288133481 learned_addr learned my addr 172.21.15.98:0/4288133481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.518+0000 7fbaae332700 1 -- 172.21.15.98:0/4288133481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa8197430 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fbaae332700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004f80 tx=0x7fbaa4005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaa4026070 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbaa400b6f0 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa8197720 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaa40037f0 con 0x7fbaa8100e60 2024-04-26T04:20:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.519+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa819b7c0 con 0x7fbaa8100e60 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fbaa4029070 con 0x7fbaa8100e60 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 0x7fba9403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fbaa401e070 con 0x7fbaa8100e60 2024-04-26T04:20:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa8102340 con 0x7fbaa8100e60 2024-04-26T04:20:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fbaadb31700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 0x7fba9403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.520+0000 7fbaadb31700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 0x7fba9403b640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba98006fd0 tx=0x7fba98006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.525+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbaa4030070 con 0x7fbaa8100e60 2024-04-26T04:20:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.722+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbaa8063850 con 0x7fbaa8100e60 2024-04-26T04:20:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.722+0000 7fba9f7fe700 1 -- 172.21.15.98:0/4288133481 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbaa4013070 con 0x7fbaa8100e60 2024-04-26T04:20:34.724 INFO:teuthology.orchestra.run.smithi098.stdout:55834574865 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.724+0000 7fbab0596700 10 mgrc shutdown 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 msgr2=0x7fba9403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 0x7fba9403b640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fba98006fd0 tx=0x7fba98006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 msgr2=0x7fbaa8196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4004f80 tx=0x7fbaa4005060 comp rx=0 tx=0).stop 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 shutdown_connections 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fba94039180 0x7fba9403b640 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 --2- 172.21.15.98:0/4288133481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa8100e60 0x7fbaa8196ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 >> 172.21.15.98:0/4288133481 conn(0x7fbaa80fc730 msgr2=0x7fbaa806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 shutdown_connections 2024-04-26T04:20:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:34.725+0000 7fbab0596700 1 -- 172.21.15.98:0/4288133481 wait complete. 2024-04-26T04:20:34.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:34 smithi098 ceph-mon[27184]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:35.372 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574865 for osd.1 2024-04-26T04:20:35.373 DEBUG:teuthology.parallel:result is None 2024-04-26T04:20:35.373 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T04:20:35.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T04:20:35.607 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:35.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:35 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/4288133481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T04:20:36.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:36 smithi098 ceph-mon[27184]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.155+0000 7fcc72a1c700 1 -- 172.21.15.98:0/2691122594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 msgr2=0x7fcc6c102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.155+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/2691122594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c102350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009910 tx=0x7fcc5c009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.155+0000 7fcc72a1c700 1 -- 172.21.15.98:0/2691122594 shutdown_connections 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.155+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/2691122594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c102350 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.155+0000 7fcc72a1c700 1 -- 172.21.15.98:0/2691122594 >> 172.21.15.98:0/2691122594 conn(0x7fcc6c0fd800 msgr2=0x7fcc6c0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 -- 172.21.15.98:0/2691122594 shutdown_connections 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 -- 172.21.15.98:0/2691122594 wait complete. 2024-04-26T04:20:37.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 Processor -- start 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 -- start start 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc72a1c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c10b840 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.156+0000 7fcc63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32912/0 (socket says 172.21.15.98:32912) 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc63fff700 1 -- 172.21.15.98:0/626098974 learned_addr learned my addr 172.21.15.98:0/626098974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc63fff700 1 -- 172.21.15.98:0/626098974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c197420 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc63fff700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c004f80 tx=0x7fcc5c005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc5c026070 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc5c00b6f0 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc5c0037f0 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6c197710 con 0x7fcc6c101f70 2024-04-26T04:20:37.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.157+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6c19b7b0 con 0x7fcc6c101f70 2024-04-26T04:20:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fcc5c029070 con 0x7fcc6c101f70 2024-04-26T04:20:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:37.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6c103450 con 0x7fcc6c101f70 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 0x7fcc5803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.158+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fcc5c01e070 con 0x7fcc6c101f70 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.159+0000 7fcc6bfff700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 0x7fcc5803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:37.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.159+0000 7fcc6bfff700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 0x7fcc5803b690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc54006fd0 tx=0x7fcc54006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:37.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.163+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc5c013070 con 0x7fcc6c101f70 2024-04-26T04:20:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.354+0000 7fcc72a1c700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:20:37.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.354+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc6c063850 con 0x7fcc580391d0 2024-04-26T04:20:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.355+0000 7fcc69ffb700 1 -- 172.21.15.98:0/626098974 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+9966 (secure 0 0 0) 0x7fcc6c063850 con 0x7fcc580391d0 2024-04-26T04:20:37.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.355+0000 7fcc69ffb700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:20:37.356 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 10 mgrc shutdown 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 msgr2=0x7fcc5803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 0x7fcc5803b690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc54006fd0 tx=0x7fcc54006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 msgr2=0x7fcc6c196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c004f80 tx=0x7fcc5c005060 comp rx=0 tx=0).stop 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 shutdown_connections 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcc580391d0 0x7fcc5803b690 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 --2- 172.21.15.98:0/626098974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc6c101f70 0x7fcc6c196ee0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 >> 172.21.15.98:0/626098974 conn(0x7fcc6c0fd800 msgr2=0x7fcc6c109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:37.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 shutdown_connections 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:37.357+0000 7fcc72a1c700 1 -- 172.21.15.98:0/626098974 wait complete. 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-26T04:20:38.013 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-04-26T04:20:35.554058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281235456,"kb_used":94260,"kb_used_data":696,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281141196,"statfs":{"total":287985106944,"available":287888584704,"internally_reserved":0,"allocated":712704,"data_stored":1929336,"data_compressed":11052,"data_compressed_allocated":110592,"data_compressed_original":1720320,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002109"},"pg_stats":[{"pgid":"1.0","version":"21'97","reported_seq":140,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-26T04:20:05.516726+0000","last_change":"2024-04-26T04:20:03.331413+0000","last_active":"2024-04-26T04:20:05.516726+0000","last_peered":"2024-04-26T04:20:05.516726+0000","last_clean":"2024-04-26T04:20:05.516726+0000","last_became_active":"2024-04-26T04:20:03.330188+0000","last_became_peered":"2024-04-26T04:20:03.330188+0000","last_unstale":"2024-04-26T04:20:05.516726+0000","last_undegraded":"2024-04-26T04:20:05.516726+0000","last_fullsized":"2024-04-26T04:20:05.516726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:20:01.836033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:20:01.836033+0000","last_clean_scrub_stamp":"2024-04-26T04:20:01.836033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:46:04.608856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1771104,"data_compressed":11052,"data_compressed_allocated":110592,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":35580,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709572,"statfs":{"total":95995035648,"available":95958601728,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738389,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717892,"statfs":{"total":95995035648,"available":95967121408,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713732,"statfs":{"total":95995035648,"available":95962861568,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:20:38.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T04:20:38.248 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:38 smithi098 ceph-mon[27184]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:20:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:38 smithi098 ceph-mon[27184]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/293690538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 msgr2=0x7fa9b8100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/293690538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8100a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009910 tx=0x7fa9ac009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/293690538 shutdown_connections 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/293690538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8100a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/293690538 >> 172.21.15.98:0/293690538 conn(0x7fa9b8077030 msgr2=0x7fa9b8077440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/293690538 shutdown_connections 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.820+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/293690538 wait complete. 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9bd5a1700 1 Processor -- start 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9bd5a1700 1 -- start start 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9bd5a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9bd5a1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b810b820 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9b6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60822/0 (socket says 172.21.15.98:60822) 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9b6ffd700 1 -- 172.21.15.98:0/2417234187 learned_addr learned my addr 172.21.15.98:0/2417234187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.821+0000 7fa9b6ffd700 1 -- 172.21.15.98:0/2417234187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b81973d0 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa9b6ffd700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac006010 tx=0x7fa9ac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9ac026070 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa9ac00b6f0 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b81976c0 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9ac0037f0 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.822+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b819b710 con 0x7fa9b8100620 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b806a150 con 0x7fa9b8100620 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fa9ac029070 con 0x7fa9b8100620 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 0x7fa9a003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fa9ac01e070 con 0x7fa9b8100620 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.823+0000 7fa9b67fc700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 0x7fa9a003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.824+0000 7fa9b67fc700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 0x7fa9a003b640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8006fd0 tx=0x7fa9a8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:39.828+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9ac013070 con 0x7fa9b8100620 2024-04-26T04:20:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.015+0000 7fa9bd5a1700 20 mgrc start_command cmd: [{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:20:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.015+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9b8063850 con 0x7fa9a0039180 2024-04-26T04:20:40.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.016+0000 7fa99ffff700 1 -- 172.21.15.98:0/2417234187 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+9966 (secure 0 0 0) 0x7fa9b8063850 con 0x7fa9a0039180 2024-04-26T04:20:40.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.016+0000 7fa99ffff700 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:20:40.017 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:40.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 10 mgrc shutdown 2024-04-26T04:20:40.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 msgr2=0x7fa9a003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 0x7fa9a003b640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8006fd0 tx=0x7fa9a8006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 msgr2=0x7fa9b8196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac006010 tx=0x7fa9ac005030 comp rx=0 tx=0).stop 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 shutdown_connections 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fa9a0039180 0x7fa9a003b640 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 --2- 172.21.15.98:0/2417234187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9b8100620 0x7fa9b8196e90 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac006010 tx=0x7fa9ac005030 comp rx=0 tx=0).stop 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 >> 172.21.15.98:0/2417234187 conn(0x7fa9b8077030 msgr2=0x7fa9b80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 shutdown_connections 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:40.018+0000 7fa9bd5a1700 1 -- 172.21.15.98:0/2417234187 wait complete. 2024-04-26T04:20:40.020 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-26T04:20:40.660 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:40 smithi098 ceph-mon[27184]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:40.662 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-04-26T04:20:39.554761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281235456,"kb_used":94260,"kb_used_data":696,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281141196,"statfs":{"total":287985106944,"available":287888584704,"internally_reserved":0,"allocated":712704,"data_stored":1929336,"data_compressed":11052,"data_compressed_allocated":110592,"data_compressed_original":1720320,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002266"},"pg_stats":[{"pgid":"1.0","version":"21'97","reported_seq":140,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-26T04:20:05.516726+0000","last_change":"2024-04-26T04:20:03.331413+0000","last_active":"2024-04-26T04:20:05.516726+0000","last_peered":"2024-04-26T04:20:05.516726+0000","last_clean":"2024-04-26T04:20:05.516726+0000","last_became_active":"2024-04-26T04:20:03.330188+0000","last_became_peered":"2024-04-26T04:20:03.330188+0000","last_unstale":"2024-04-26T04:20:05.516726+0000","last_undegraded":"2024-04-26T04:20:05.516726+0000","last_fullsized":"2024-04-26T04:20:05.516726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:20:01.836033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:20:01.836033+0000","last_clean_scrub_stamp":"2024-04-26T04:20:01.836033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:46:04.608856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1771104,"data_compressed":11052,"data_compressed_allocated":110592,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":35580,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709572,"statfs":{"total":95995035648,"available":95958601728,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738390,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717892,"statfs":{"total":95995035648,"available":95967121408,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713732,"statfs":{"total":95995035648,"available":95962861568,"internally_reserved":0,"allocated":237568,"data_stored":643112,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3684,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:20:40.662 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T04:20:40.663 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T04:20:40.663 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T04:20:40.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448 shell --fsid b735c86e-0383-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T04:20:40.897 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T04:20:41.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:41 smithi098 ceph-mon[27184]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:20:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 -- 172.21.15.98:0/2148171167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 msgr2=0x7fcf40100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 --2- 172.21.15.98:0/2148171167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40100570 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009910 tx=0x7fcf34009c20 comp rx=0 tx=0).stop 2024-04-26T04:20:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 -- 172.21.15.98:0/2148171167 shutdown_connections 2024-04-26T04:20:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 --2- 172.21.15.98:0/2148171167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40100570 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:42.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 -- 172.21.15.98:0/2148171167 >> 172.21.15.98:0/2148171167 conn(0x7fcf400fba20 msgr2=0x7fcf400fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 -- 172.21.15.98:0/2148171167 shutdown_connections 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.502+0000 7fcf46658700 1 -- 172.21.15.98:0/2148171167 wait complete. 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf46658700 1 Processor -- start 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf46658700 1 -- start start 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf46658700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf46658700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcf40109ea0 con 0x7fcf40100190 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60836/0 (socket says 172.21.15.98:60836) 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.503+0000 7fcf3ffff700 1 -- 172.21.15.98:0/1725273857 learned_addr learned my addr 172.21.15.98:0/1725273857 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:20:42.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf3ffff700 1 -- 172.21.15.98:0/1725273857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf40198280 con 0x7fcf40100190 2024-04-26T04:20:42.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf3ffff700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcf34004f80 tx=0x7fcf34005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf34026070 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcf3400b6f0 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf40198570 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf340037f0 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.504+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf40198a30 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fcf34029070 con 0x7fcf40100190 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 0x7fcf2803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:42.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fcf3401e070 con 0x7fcf40100190 2024-04-26T04:20:42.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.505+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf40101670 con 0x7fcf40100190 2024-04-26T04:20:42.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.506+0000 7fcf3f7fe700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 0x7fcf2803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:42.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.506+0000 7fcf3f7fe700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 0x7fcf2803b690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf30006fd0 tx=0x7fcf30006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:42.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.511+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcf34030070 con 0x7fcf40100190 2024-04-26T04:20:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.753+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcf40063850 con 0x7fcf40100190 2024-04-26T04:20:42.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.753+0000 7fcf3d7fa700 1 -- 172.21.15.98:0/1725273857 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fcf34013070 con 0x7fcf40100190 2024-04-26T04:20:42.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-26T04:20:42.756 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T04:20:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 10 mgrc shutdown 2024-04-26T04:20:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 msgr2=0x7fcf2803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:42.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 0x7fcf2803b690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf30006fd0 tx=0x7fcf30006e40 comp rx=0 tx=0).stop 2024-04-26T04:20:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 msgr2=0x7fcf40197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:42.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcf34004f80 tx=0x7fcf34005060 comp rx=0 tx=0).stop 2024-04-26T04:20:42.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:42 smithi098 ceph-mon[27184]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 shutdown_connections 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fcf280391d0 0x7fcf2803b690 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.755+0000 7fcf46658700 1 --2- 172.21.15.98:0/1725273857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf40100190 0x7fcf40197d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.756+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 >> 172.21.15.98:0/1725273857 conn(0x7fcf400fba20 msgr2=0x7fcf400fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.756+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 shutdown_connections 2024-04-26T04:20:42.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-26T04:20:42.756+0000 7fcf46658700 1 -- 172.21.15.98:0/1725273857 wait complete. 2024-04-26T04:20:43.363 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T04:20:43.363 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T04:20:43.364 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T04:20:43.377 INFO:tasks.workunit:Pulling workunits from ref ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:20:43.378 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T04:20:43.378 DEBUG:teuthology.orchestra.run.smithi165:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-26T04:20:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:20:43.397 INFO:teuthology.orchestra.run.smithi165.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-26T04:20:43.398 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -- /home/ubuntu/cephtest/mnt.1 2024-04-26T04:20:43.455 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.1 2024-04-26T04:20:43.456 DEBUG:teuthology.orchestra.run.smithi165:> cd -- /home/ubuntu/cephtest/mnt.1 && mkdir -- client.1 2024-04-26T04:20:43.513 INFO:tasks.workunit:timeout=3h 2024-04-26T04:20:43.513 INFO:tasks.workunit:cleanup=True 2024-04-26T04:20:43.513 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:20:43.570 INFO:tasks.workunit.client.1.smithi165.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-26T04:20:44.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:43 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/1725273857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T04:20:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:44 smithi098 ceph-mon[27184]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:46 smithi098 ceph-mon[27184]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:48 smithi098 ceph-mon[27184]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:50 smithi098 ceph-mon[27184]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:53.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:52 smithi098 ceph-mon[27184]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:54 smithi098 ceph-mon[27184]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:56 smithi098 ceph-mon[27184]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:20:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:58 smithi098 ceph-mon[27184]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:00.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:20:59 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:21:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:00 smithi098 ceph-mon[27184]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:21:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:21:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:21:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:02 smithi098 ceph-mon[27184]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:04 smithi098 ceph-mon[27184]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:06 smithi098 ceph-mon[27184]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:08 smithi098 ceph-mon[27184]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:10 smithi098 ceph-mon[27184]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:12 smithi098 ceph-mon[27184]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:14 smithi098 ceph-mon[27184]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:16 smithi098 ceph-mon[27184]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:18 smithi098 ceph-mon[27184]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:20 smithi098 ceph-mon[27184]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:22 smithi098 ceph-mon[27184]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:24 smithi098 ceph-mon[27184]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:26 smithi098 ceph-mon[27184]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:28 smithi098 ceph-mon[27184]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:29.731 INFO:tasks.workunit.client.1.smithi165.stderr:Note: switching to 'ddeaa91797339a19172c3036ff48cca58c12f448'. 2024-04-26T04:21:29.731 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.731 INFO:tasks.workunit.client.1.smithi165.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr: git switch -c 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.732 INFO:tasks.workunit.client.1.smithi165.stderr:Or undo this operation with: 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr: git switch - 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr: 2024-04-26T04:21:29.733 INFO:tasks.workunit.client.1.smithi165.stderr:HEAD is now at ddeaa917973 Merge branch 'wip-rf-56389-to-squid' of https://github.com/ronen-fr/ceph into wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:21:29.738 DEBUG:teuthology.orchestra.run.smithi165:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-26T04:21:29.854 INFO:tasks.workunit.client.1.smithi165.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T04:21:29.856 INFO:tasks.workunit.client.1.smithi165.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-26T04:21:29.856 INFO:tasks.workunit.client.1.smithi165.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T04:21:30.511 INFO:tasks.workunit.client.1.smithi165.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T04:21:30.571 INFO:tasks.workunit.client.1.smithi165.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T04:21:30.612 INFO:tasks.workunit.client.1.smithi165.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-26T04:21:30.614 INFO:tasks.workunit.client.1.smithi165.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-26T04:21:30.614 INFO:tasks.workunit.client.1.smithi165.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T04:21:30.653 INFO:tasks.workunit.client.1.smithi165.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-26T04:21:30.656 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-26T04:21:30.656 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-26T04:21:30.672 INFO:tasks.workunit:Running workunits matching cephadm/create_iscsi_disks.sh on client.1... 2024-04-26T04:21:30.673 INFO:tasks.workunit:Running workunit cephadm/create_iscsi_disks.sh... 2024-04-26T04:21:30.673 DEBUG:teuthology.orchestra.run.smithi165:workunit test cephadm/create_iscsi_disks.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh 2024-04-26T04:21:30.744 INFO:tasks.workunit.client.1.smithi165.stderr:+ grep -q rhel /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:21:30.745 INFO:tasks.workunit.client.1.smithi165.stderr:+ '[' -z '' ']' 2024-04-26T04:21:30.745 INFO:tasks.workunit.client.1.smithi165.stderr:+ SUDO=sudo 2024-04-26T04:21:30.745 INFO:tasks.workunit.client.1.smithi165.stderr:+ DISK_FILE_SIZE=16106127360 2024-04-26T04:21:30.745 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo yum install -y targetcli iscsi-initiator-utils 2024-04-26T04:21:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:30 smithi098 ceph-mon[27184]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:31.245 INFO:tasks.workunit.client.1.smithi165.stdout:Last metadata expiration check: 0:07:13 ago on Fri 26 Apr 2024 04:14:18 AM UTC. 2024-04-26T04:21:31.367 INFO:tasks.workunit.client.1.smithi165.stdout:Dependencies resolved. 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout:================================================================================ 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout: Package Arch Version Repo Size 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout:================================================================================ 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout:Installing: 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout: iscsi-initiator-utils x86_64 6.2.1.9-1.gita65a472.el9 baseos 391 k 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout: targetcli noarch 2.1.57-2.el9 appstream 75 k 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout:Installing dependencies: 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout: iscsi-initiator-utils-iscsiuio x86_64 6.2.1.9-1.gita65a472.el9 baseos 81 k 2024-04-26T04:21:31.368 INFO:tasks.workunit.client.1.smithi165.stdout: isns-utils-libs x86_64 0.101-4.el9 baseos 101 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: python3-rtslib noarch 2.1.76-1.el9 appstream 100 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: target-restore noarch 2.1.76-1.el9 appstream 14 k 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout:Transaction Summary 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout:================================================================================ 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout:Install 9 Packages 2024-04-26T04:21:31.369 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:31.370 INFO:tasks.workunit.client.1.smithi165.stdout:Total download size: 1.7 M 2024-04-26T04:21:31.370 INFO:tasks.workunit.client.1.smithi165.stdout:Installed size: 6.5 M 2024-04-26T04:21:31.370 INFO:tasks.workunit.client.1.smithi165.stdout:Downloading Packages: 2024-04-26T04:21:31.862 INFO:tasks.workunit.client.1.smithi165.stdout:(1/9): iscsi-initiator-utils-iscsiuio-6.2.1.9-1 317 kB/s | 81 kB 00:00 2024-04-26T04:21:31.886 INFO:tasks.workunit.client.1.smithi165.stdout:(2/9): iscsi-initiator-utils-6.2.1.9-1.gita65a4 1.4 MB/s | 391 kB 00:00 2024-04-26T04:21:31.911 INFO:tasks.workunit.client.1.smithi165.stdout:(3/9): isns-utils-libs-0.101-4.el9.x86_64.rpm 330 kB/s | 101 kB 00:00 2024-04-26T04:21:31.937 INFO:tasks.workunit.client.1.smithi165.stdout:(4/9): python3-configshell-1.1.30-1.el9.noarch. 970 kB/s | 72 kB 00:00 2024-04-26T04:21:31.961 INFO:tasks.workunit.client.1.smithi165.stdout:(5/9): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-04-26T04:21:32.120 INFO:tasks.workunit.client.1.smithi165.stdout:(6/9): python3-urwid-2.1.2-4.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2024-04-26T04:21:32.145 INFO:tasks.workunit.client.1.smithi165.stdout:(7/9): python3-rtslib-2.1.76-1.el9.noarch.rpm 480 kB/s | 100 kB 00:00 2024-04-26T04:21:32.162 INFO:tasks.workunit.client.1.smithi165.stdout:(8/9): target-restore-2.1.76-1.el9.noarch.rpm 72 kB/s | 14 kB 00:00 2024-04-26T04:21:32.187 INFO:tasks.workunit.client.1.smithi165.stdout:(9/9): targetcli-2.1.57-2.el9.noarch.rpm 1.1 MB/s | 75 kB 00:00 2024-04-26T04:21:32.190 INFO:tasks.workunit.client.1.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:21:32.190 INFO:tasks.workunit.client.1.smithi165.stdout:Total 2.1 MB/s | 1.7 MB 00:00 2024-04-26T04:21:32.286 INFO:tasks.workunit.client.1.smithi165.stdout:Running transaction check 2024-04-26T04:21:32.299 INFO:tasks.workunit.client.1.smithi165.stdout:Transaction check succeeded. 2024-04-26T04:21:32.299 INFO:tasks.workunit.client.1.smithi165.stdout:Running transaction test 2024-04-26T04:21:32.411 INFO:tasks.workunit.client.1.smithi165.stdout:Transaction test succeeded. 2024-04-26T04:21:32.411 INFO:tasks.workunit.client.1.smithi165.stdout:Running transaction 2024-04-26T04:21:32.944 INFO:tasks.workunit.client.1.smithi165.stdout: Preparing : 1/1 2024-04-26T04:21:33.024 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/9 2024-04-26T04:21:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:32 smithi098 ceph-mon[27184]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:33.057 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/9 2024-04-26T04:21:33.112 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/9 2024-04-26T04:21:33.146 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : python3-rtslib-2.1.76-1.el9.noarch 4/9 2024-04-26T04:21:33.184 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : target-restore-2.1.76-1.el9.noarch 5/9 2024-04-26T04:21:33.204 INFO:tasks.workunit.client.1.smithi165.stdout: Running scriptlet: target-restore-2.1.76-1.el9.noarch 5/9 2024-04-26T04:21:33.230 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : isns-utils-libs-0.101-4.el9.x86_64 6/9 2024-04-26T04:21:33.260 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : iscsi-initiator-utils-iscsiuio-6.2.1.9-1.gita65a472. 7/9 2024-04-26T04:21:33.274 INFO:tasks.workunit.client.1.smithi165.stdout: Running scriptlet: iscsi-initiator-utils-iscsiuio-6.2.1.9-1.gita65a472. 7/9 2024-04-26T04:21:33.274 INFO:tasks.workunit.client.1.smithi165.stdout:Created symlink /etc/systemd/system/sockets.target.wants/iscsiuio.socket → /usr/lib/systemd/system/iscsiuio.socket. 2024-04-26T04:21:33.274 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:33.320 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_6 8/9 2024-04-26T04:21:33.337 INFO:tasks.workunit.client.1.smithi165.stdout: Running scriptlet: iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_6 8/9 2024-04-26T04:21:33.337 INFO:tasks.workunit.client.1.smithi165.stdout:Created symlink /etc/systemd/system/sysinit.target.wants/iscsi-starter.service → /usr/lib/systemd/system/iscsi-starter.service. 2024-04-26T04:21:33.337 INFO:tasks.workunit.client.1.smithi165.stdout:Created symlink /etc/systemd/system/sockets.target.wants/iscsid.socket → /usr/lib/systemd/system/iscsid.socket. 2024-04-26T04:21:33.337 INFO:tasks.workunit.client.1.smithi165.stdout:Created symlink /etc/systemd/system/sysinit.target.wants/iscsi-onboot.service → /usr/lib/systemd/system/iscsi-onboot.service. 2024-04-26T04:21:33.337 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:33.417 INFO:tasks.workunit.client.1.smithi165.stdout: Installing : targetcli-2.1.57-2.el9.noarch 9/9 2024-04-26T04:21:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:34 smithi098 ceph-mon[27184]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:36.430 INFO:tasks.workunit.client.1.smithi165.stdout: Running scriptlet: targetcli-2.1.57-2.el9.noarch 9/9 2024-04-26T04:21:36.430 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_6 1/9 2024-04-26T04:21:36.430 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : iscsi-initiator-utils-iscsiuio-6.2.1.9-1.gita65a472. 2/9 2024-04-26T04:21:36.430 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : isns-utils-libs-0.101-4.el9.x86_64 3/9 2024-04-26T04:21:36.431 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/9 2024-04-26T04:21:36.431 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/9 2024-04-26T04:21:36.431 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/9 2024-04-26T04:21:36.431 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : python3-rtslib-2.1.76-1.el9.noarch 7/9 2024-04-26T04:21:36.431 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : target-restore-2.1.76-1.el9.noarch 8/9 2024-04-26T04:21:36.905 INFO:tasks.workunit.client.1.smithi165.stdout: Verifying : targetcli-2.1.57-2.el9.noarch 9/9 2024-04-26T04:21:36.905 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:36.905 INFO:tasks.workunit.client.1.smithi165.stdout:Installed: 2024-04-26T04:21:36.905 INFO:tasks.workunit.client.1.smithi165.stdout: iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_64 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: iscsi-initiator-utils-iscsiuio-6.2.1.9-1.gita65a472.el9.x86_64 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: isns-utils-libs-0.101-4.el9.x86_64 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: python3-configshell-1:1.1.30-1.el9.noarch 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: python3-kmod-0.9-32.el9.x86_64 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: python3-rtslib-2.1.76-1.el9.noarch 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: python3-urwid-2.1.2-4.el9.x86_64 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: target-restore-2.1.76-1.el9.noarch 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: targetcli-2.1.57-2.el9.noarch 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout: 2024-04-26T04:21:36.906 INFO:tasks.workunit.client.1.smithi165.stdout:Complete! 2024-04-26T04:21:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:36 smithi098 ceph-mon[27184]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:37.340 INFO:tasks.workunit.client.1.smithi165.stderr:++ hostname 2024-04-26T04:21:37.341 INFO:tasks.workunit.client.1.smithi165.stderr:+ TARGET_NAME=iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar 2024-04-26T04:21:37.341 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi create iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar 2024-04-26T04:21:37.689 INFO:tasks.workunit.client.1.smithi165.stdout:Created target iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar. 2024-04-26T04:21:37.689 INFO:tasks.workunit.client.1.smithi165.stdout:Created TPG 1. 2024-04-26T04:21:37.690 INFO:tasks.workunit.client.1.smithi165.stdout:Global pref auto_add_default_portal=true 2024-04-26T04:21:37.690 INFO:tasks.workunit.client.1.smithi165.stdout:Created default portal listening on all IPs (0.0.0.0), port 3260. 2024-04-26T04:21:37.706 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1/portals delete 0.0.0.0 3260 2024-04-26T04:21:37.817 INFO:tasks.workunit.client.1.smithi165.stdout:Deleted network portal 0.0.0.0:3260 2024-04-26T04:21:37.834 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1/portals create 127.0.0.1 3260 2024-04-26T04:21:37.948 INFO:tasks.workunit.client.1.smithi165.stdout:Using default IP port 3260 2024-04-26T04:21:37.948 INFO:tasks.workunit.client.1.smithi165.stdout:Created network portal 127.0.0.1:3260. 2024-04-26T04:21:37.964 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1 set attribute generate_node_acls=1 2024-04-26T04:21:38.077 INFO:tasks.workunit.client.1.smithi165.stdout:Parameter generate_node_acls is now '1'. 2024-04-26T04:21:38.094 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1 set attribute demo_mode_write_protect=0 2024-04-26T04:21:38.207 INFO:tasks.workunit.client.1.smithi165.stdout:Parameter demo_mode_write_protect is now '0'. 2024-04-26T04:21:38.226 INFO:tasks.workunit.client.1.smithi165.stderr:++ seq 3 2024-04-26T04:21:38.239 INFO:tasks.workunit.client.1.smithi165.stderr:+ for i in $(seq 3) 2024-04-26T04:21:38.239 INFO:tasks.workunit.client.1.smithi165.stderr:+ DISK_FILE=/tmp/disk1 2024-04-26T04:21:38.239 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo truncate --size 16106127360 /tmp/disk1 2024-04-26T04:21:38.255 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio create lun1 /tmp/disk1 2024-04-26T04:21:38.403 INFO:tasks.workunit.client.1.smithi165.stdout:Created fileio lun1 with size 16106127360 2024-04-26T04:21:38.419 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio/lun1 set attribute optimal_sectors=0 2024-04-26T04:21:38.535 INFO:tasks.workunit.client.1.smithi165.stdout:Parameter optimal_sectors is now '0'. 2024-04-26T04:21:38.550 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1/luns create /backstores/fileio/lun1 2024-04-26T04:21:38.682 INFO:tasks.workunit.client.1.smithi165.stdout:Created LUN 0. 2024-04-26T04:21:38.698 INFO:tasks.workunit.client.1.smithi165.stderr:+ for i in $(seq 3) 2024-04-26T04:21:38.699 INFO:tasks.workunit.client.1.smithi165.stderr:+ DISK_FILE=/tmp/disk2 2024-04-26T04:21:38.699 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo truncate --size 16106127360 /tmp/disk2 2024-04-26T04:21:38.708 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio create lun2 /tmp/disk2 2024-04-26T04:21:38.823 INFO:tasks.workunit.client.1.smithi165.stdout:Created fileio lun2 with size 16106127360 2024-04-26T04:21:38.839 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio/lun2 set attribute optimal_sectors=0 2024-04-26T04:21:38.953 INFO:tasks.workunit.client.1.smithi165.stdout:Parameter optimal_sectors is now '0'. 2024-04-26T04:21:38.968 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1/luns create /backstores/fileio/lun2 2024-04-26T04:21:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:38 smithi098 ceph-mon[27184]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:39.095 INFO:tasks.workunit.client.1.smithi165.stdout:Created LUN 1. 2024-04-26T04:21:39.111 INFO:tasks.workunit.client.1.smithi165.stderr:+ for i in $(seq 3) 2024-04-26T04:21:39.111 INFO:tasks.workunit.client.1.smithi165.stderr:+ DISK_FILE=/tmp/disk3 2024-04-26T04:21:39.111 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo truncate --size 16106127360 /tmp/disk3 2024-04-26T04:21:39.121 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio create lun3 /tmp/disk3 2024-04-26T04:21:39.236 INFO:tasks.workunit.client.1.smithi165.stdout:Created fileio lun3 with size 16106127360 2024-04-26T04:21:39.251 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /backstores/fileio/lun3 set attribute optimal_sectors=0 2024-04-26T04:21:39.364 INFO:tasks.workunit.client.1.smithi165.stdout:Parameter optimal_sectors is now '0'. 2024-04-26T04:21:39.380 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo targetcli /iscsi/iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar/tpg1/luns create /backstores/fileio/lun3 2024-04-26T04:21:39.509 INFO:tasks.workunit.client.1.smithi165.stdout:Created LUN 2. 2024-04-26T04:21:39.525 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo iscsiadm -m discovery -t sendtargets -p 127.0.0.1 2024-04-26T04:21:40.603 INFO:tasks.workunit.client.1.smithi165.stdout:127.0.0.1:3260,1 iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar 2024-04-26T04:21:40.604 INFO:tasks.workunit.client.1.smithi165.stderr:+ sudo iscsiadm -m node -p 127.0.0.1 -T iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar -l 2024-04-26T04:21:40.629 INFO:tasks.workunit.client.1.smithi165.stdout:Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar, portal: 127.0.0.1,3260] 2024-04-26T04:21:40.630 INFO:tasks.workunit.client.1.smithi165.stdout:Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.smithi165.x8664:sn.foobar, portal: 127.0.0.1,3260] successful. 2024-04-26T04:21:40.632 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-26T04:21:40.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-26T04:21:40.697 INFO:tasks.workunit:Stopping ['cephadm/create_iscsi_disks.sh'] on client.1... 2024-04-26T04:21:40.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-26T04:21:41.017 DEBUG:teuthology.parallel:result is None 2024-04-26T04:21:41.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-26T04:21:41.041 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-26T04:21:41.041 DEBUG:teuthology.orchestra.run.smithi165:> rmdir -- /home/ubuntu/cephtest/mnt.1 2024-04-26T04:21:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:40 smithi098 ceph-mon[27184]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:41.105 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.1/client.1 2024-04-26T04:21:41.105 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T04:21:41.116 INFO:tasks.workunit:Pulling workunits from ref ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:21:41.116 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T04:21:41.116 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T04:21:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:21:41.136 INFO:teuthology.orchestra.run.smithi098.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-26T04:21:41.136 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-26T04:21:41.194 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-26T04:21:41.194 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-26T04:21:41.251 INFO:tasks.workunit:timeout=3h 2024-04-26T04:21:41.251 INFO:tasks.workunit:cleanup=True 2024-04-26T04:21:41.252 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:21:41.309 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T04:21:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:42 smithi098 ceph-mon[27184]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:44 smithi098 ceph-mon[27184]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:46 smithi098 ceph-mon[27184]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:48 smithi098 ceph-mon[27184]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:50 smithi098 ceph-mon[27184]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:53.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:52 smithi098 ceph-mon[27184]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:54 smithi098 ceph-mon[27184]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:56 smithi098 ceph-mon[27184]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:21:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:21:58 smithi098 ceph-mon[27184]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:00 smithi098 ceph-mon[27184]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:22:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:22:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:22:01.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:22:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:02 smithi098 ceph-mon[27184]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:04 smithi098 ceph-mon[27184]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:06 smithi098 ceph-mon[27184]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:08 smithi098 ceph-mon[27184]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:10 smithi098 ceph-mon[27184]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:12 smithi098 ceph-mon[27184]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:14 smithi098 ceph-mon[27184]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:16 smithi098 ceph-mon[27184]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:18 smithi098 ceph-mon[27184]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:20 smithi098 ceph-mon[27184]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:22 smithi098 ceph-mon[27184]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:24 smithi098 ceph-mon[27184]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:26 smithi098 ceph-mon[27184]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:28 smithi098 ceph-mon[27184]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:30 smithi098 ceph-mon[27184]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:32 smithi098 ceph-mon[27184]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:34 smithi098 ceph-mon[27184]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:36 smithi098 ceph-mon[27184]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:39.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:38 smithi098 ceph-mon[27184]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:40 smithi098 ceph-mon[27184]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:42 smithi098 ceph-mon[27184]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:44.932 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to 'ddeaa91797339a19172c3036ff48cca58c12f448'. 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-04-26T04:22:44.933 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.934 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-04-26T04:22:44.934 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.934 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T04:22:44.934 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:22:44.934 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at ddeaa917973 Merge branch 'wip-rf-56389-to-squid' of https://github.com/ronen-fr/ceph into wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:22:44.938 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T04:22:45.002 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T04:22:45.004 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T04:22:45.004 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T04:22:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:44 smithi098 ceph-mon[27184]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:45.743 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T04:22:45.799 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T04:22:45.841 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T04:22:45.842 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T04:22:45.842 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T04:22:45.881 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T04:22:45.885 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-26T04:22:45.885 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T04:22:45.941 INFO:tasks.workunit:Running workunits matching cephadm/test_dashboard_e2e.sh on client.0... 2024-04-26T04:22:45.942 INFO:tasks.workunit:Running workunit cephadm/test_dashboard_e2e.sh... 2024-04-26T04:22:45.943 DEBUG:teuthology.orchestra.run.smithi098:workunit test cephadm/test_dashboard_e2e.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh 2024-04-26T04:22:46.008 INFO:tasks.workunit.client.0.smithi098.stderr:+++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh 2024-04-26T04:22:46.009 INFO:tasks.workunit.client.0.smithi098.stderr:++ cd /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm 2024-04-26T04:22:46.009 INFO:tasks.workunit.client.0.smithi098.stderr:++ pwd 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ SCRIPT_DIR=/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ DASHBOARD_FRONTEND_DIR=/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/../../../src/pybind/mgr/dashboard/frontend 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' -z '' ']' 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ SUDO=sudo 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ install_common 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ NODEJS_VERSION=16 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q debian /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:22:46.010 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q rhel /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:22:46.011 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q 'CentOS Stream 9' /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:22:46.012 INFO:tasks.workunit.client.0.smithi098.stderr:+ NODEJS_VERSION=18 2024-04-26T04:22:46.012 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo yum module -y enable nodejs:18 2024-04-26T04:22:46.512 INFO:tasks.workunit.client.0.smithi098.stdout:Last metadata expiration check: 0:08:29 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:22:46.640 INFO:tasks.workunit.client.0.smithi098.stdout:Dependencies resolved. 2024-04-26T04:22:46.641 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:22:46.641 INFO:tasks.workunit.client.0.smithi098.stdout: Package Architecture Version Repository Size 2024-04-26T04:22:46.641 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:22:46.641 INFO:tasks.workunit.client.0.smithi098.stdout:Enabling module streams: 2024-04-26T04:22:46.641 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs 18 2024-04-26T04:22:46.642 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:46.642 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction Summary 2024-04-26T04:22:46.642 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:22:46.642 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:46.933 INFO:tasks.workunit.client.0.smithi098.stdout:Complete! 2024-04-26T04:22:47.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:46 smithi098 ceph-mon[27184]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:47.000 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo yum install -y jq npm 2024-04-26T04:22:47.502 INFO:tasks.workunit.client.0.smithi098.stdout:Last metadata expiration check: 0:08:30 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:22:47.583 INFO:tasks.workunit.client.0.smithi098.stdout:Package jq-1.6-16.el9.x86_64 is already installed. 2024-04-26T04:22:47.627 INFO:tasks.workunit.client.0.smithi098.stdout:Dependencies resolved. 2024-04-26T04:22:47.627 INFO:tasks.workunit.client.0.smithi098.stdout:========================================================================================= 2024-04-26T04:22:47.627 INFO:tasks.workunit.client.0.smithi098.stdout: Package Arch Version Repo Size 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:========================================================================================= 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:Installing: 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: npm x86_64 1:10.2.4-1.18.19.1.1.module_el9+986+cd5c1200 appstream 2.5 M 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:Installing dependencies: 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs x86_64 1:18.19.1-1.module_el9+986+cd5c1200 appstream 13 M 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:Installing weak dependencies: 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs-docs noarch 1:18.19.1-1.module_el9+986+cd5c1200 appstream 7.9 M 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs-full-i18n x86_64 1:18.19.1-1.module_el9+986+cd5c1200 appstream 8.5 M 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction Summary 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:========================================================================================= 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout:Install 4 Packages 2024-04-26T04:22:47.628 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:47.629 INFO:tasks.workunit.client.0.smithi098.stdout:Total download size: 31 M 2024-04-26T04:22:47.629 INFO:tasks.workunit.client.0.smithi098.stdout:Installed size: 169 M 2024-04-26T04:22:47.629 INFO:tasks.workunit.client.0.smithi098.stdout:Downloading Packages: 2024-04-26T04:22:48.417 INFO:tasks.workunit.client.0.smithi098.stdout:(1/4): nodejs-docs-18.19.1-1.module_el9+986+cd5 11 MB/s | 7.9 MB 00:00 2024-04-26T04:22:48.526 INFO:tasks.workunit.client.0.smithi098.stdout:(2/4): nodejs-full-i18n-18.19.1-1.module_el9+98 10 MB/s | 8.5 MB 00:00 2024-04-26T04:22:48.676 INFO:tasks.workunit.client.0.smithi098.stdout:(3/4): nodejs-18.19.1-1.module_el9+986+cd5c1200 13 MB/s | 13 MB 00:00 2024-04-26T04:22:48.718 INFO:tasks.workunit.client.0.smithi098.stdout:(4/4): npm-10.2.4-1.18.19.1.1.module_el9+986+cd 8.3 MB/s | 2.5 MB 00:00 2024-04-26T04:22:48.720 INFO:tasks.workunit.client.0.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:22:48.721 INFO:tasks.workunit.client.0.smithi098.stdout:Total 29 MB/s | 31 MB 00:01 2024-04-26T04:22:49.016 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction check 2024-04-26T04:22:49.034 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:22:49.034 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction test 2024-04-26T04:22:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:48 smithi098 ceph-mon[27184]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:49.394 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:22:49.450 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction 2024-04-26T04:22:50.534 INFO:tasks.workunit.client.0.smithi098.stdout: Preparing : 1/1 2024-04-26T04:22:50.766 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : nodejs-docs-1:18.19.1-1.module_el9+986+cd5c1200.noar 1/4 2024-04-26T04:22:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:50 smithi098 ceph-mon[27184]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:51.803 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : nodejs-full-i18n-1:18.19.1-1.module_el9+986+cd5c1200 2/4 2024-04-26T04:22:52.115 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : npm-1:10.2.4-1.18.19.1.1.module_el9+986+cd5c1200.x86 3/4 2024-04-26T04:22:52.196 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : nodejs-1:18.19.1-1.module_el9+986+cd5c1200.x86_64 4/4 2024-04-26T04:22:53.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:52 smithi098 ceph-mon[27184]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:54 smithi098 ceph-mon[27184]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:56.202 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: nodejs-1:18.19.1-1.module_el9+986+cd5c1200.x86_64 4/4 2024-04-26T04:22:56.202 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nodejs-1:18.19.1-1.module_el9+986+cd5c1200.x86_64 1/4 2024-04-26T04:22:56.202 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nodejs-docs-1:18.19.1-1.module_el9+986+cd5c1200.noar 2/4 2024-04-26T04:22:56.202 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nodejs-full-i18n-1:18.19.1-1.module_el9+986+cd5c1200 3/4 2024-04-26T04:22:56.514 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : npm-1:10.2.4-1.18.19.1.1.module_el9+986+cd5c1200.x86 4/4 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout:Installed: 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs-1:18.19.1-1.module_el9+986+cd5c1200.x86_64 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs-docs-1:18.19.1-1.module_el9+986+cd5c1200.noarch 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: nodejs-full-i18n-1:18.19.1-1.module_el9+986+cd5c1200.x86_64 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: npm-1:10.2.4-1.18.19.1.1.module_el9+986+cd5c1200.x86_64 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:56.515 INFO:tasks.workunit.client.0.smithi098.stdout:Complete! 2024-04-26T04:22:56.941 INFO:tasks.workunit.client.0.smithi098.stderr:+ install_chrome 2024-04-26T04:22:56.942 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q debian /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:22:56.943 INFO:tasks.workunit.client.0.smithi098.stderr:+ grep -q rhel /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release 2024-04-26T04:22:56.944 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo dd of=/etc/yum.repos.d/google-chrome.repo status=none 2024-04-26T04:22:56.955 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo yum install -y google-chrome-stable 2024-04-26T04:22:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:56 smithi098 ceph-mon[27184]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:57.692 INFO:tasks.workunit.client.0.smithi098.stdout:google-chrome 16 kB/s | 3.6 kB 00:00 2024-04-26T04:22:58.405 INFO:tasks.workunit.client.0.smithi098.stdout:Dependencies resolved. 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout:======================================================================================================== 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout:======================================================================================================== 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout:Installing: 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: google-chrome-stable x86_64 124.0.6367.91-1 google-chrome 104 M 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout:Installing dependencies: 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: ModemManager-glib x86_64 1.20.2-1.el9 baseos 335 k 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: adwaita-cursor-theme noarch 40.1.1-3.el9 appstream 650 k 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: adwaita-icon-theme noarch 40.1.1-3.el9 appstream 12 M 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: alsa-lib x86_64 1.2.10-2.el9 appstream 523 k 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-atk x86_64 2.38.0-4.el9 appstream 87 k 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-core x86_64 2.40.3-1.el9 appstream 195 k 2024-04-26T04:22:58.412 INFO:tasks.workunit.client.0.smithi098.stdout: atk x86_64 2.36.0-5.el9 appstream 292 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: avahi-glib x86_64 0.8-20.el9 appstream 15 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: bluez-libs x86_64 5.56-6.el9 baseos 85 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: bubblewrap x86_64 0.4.1-7.el9 baseos 49 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: colord-libs x86_64 1.4.5-4.el9 appstream 229 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: cups-libs x86_64 1:2.3.3op2-24.el9 baseos 262 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: exempi x86_64 2.6.0-0.2.20211007gite23c213.el9 appstream 525 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: exiv2-libs x86_64 0.27.5-2.el9 appstream 781 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: fdk-aac-free x86_64 2.0.0-8.el9 appstream 325 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: flac-libs x86_64 1.3.3-12.el9 appstream 218 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak-selinux noarch 1.12.8-1.el9 appstream 21 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak-session-helper x86_64 1.12.8-1.el9 appstream 74 k 2024-04-26T04:22:58.413 INFO:tasks.workunit.client.0.smithi098.stdout: gdk-pixbuf2-modules x86_64 2.42.6-3.el9 appstream 88 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: geoclue2 x86_64 2.6.0-7.el9 appstream 124 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: giflib x86_64 5.2.1-9.el9 appstream 49 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: graphene x86_64 1.10.6-2.el9 appstream 64 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: gsm x86_64 1.0.19-6.el9 appstream 34 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: gstreamer1 x86_64 1.22.1-2.el9 appstream 1.4 M 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: gstreamer1-plugins-base x86_64 1.22.1-2.el9 appstream 2.2 M 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: gtk-update-icon-cache x86_64 3.24.31-2.el9 appstream 35 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: gtk3 x86_64 3.24.31-2.el9 appstream 4.9 M 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: hicolor-icon-theme noarch 0.17-13.el9 appstream 219 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: iso-codes noarch 4.6.0-3.el9 appstream 3.5 M 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: jbigkit-libs x86_64 2.1-23.el9 appstream 53 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: lcms2 x86_64 2.12-3.el9 appstream 168 k 2024-04-26T04:22:58.414 INFO:tasks.workunit.client.0.smithi098.stdout: libX11-xcb x86_64 1.7.0-9.el9 appstream 11 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXcomposite x86_64 0.4.5-7.el9 appstream 24 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXcursor x86_64 1.2.0-7.el9 appstream 31 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXdamage x86_64 1.1.5-7.el9 appstream 23 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXfixes x86_64 5.0.3-16.el9 appstream 20 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXi x86_64 1.7.10-8.el9 appstream 40 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXinerama x86_64 1.1.4-10.el9 appstream 15 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXrandr x86_64 1.5.2-8.el9 appstream 28 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXtst x86_64 1.2.3-16.el9 appstream 21 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXv x86_64 1.0.11-16.el9 appstream 19 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libXxf86vm x86_64 1.1.4-18.el9 appstream 19 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libasyncns x86_64 0.8-22.el9 appstream 30 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra x86_64 0.30-27.el9 appstream 86 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libdrm x86_64 2.4.117-1.el9 appstream 161 k 2024-04-26T04:22:58.415 INFO:tasks.workunit.client.0.smithi098.stdout: libepoxy x86_64 1.5.5-4.el9 appstream 241 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-fonts noarch 1:2.1.3-5.el9 appstream 3.6 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-fonts-common noarch 1:2.1.3-5.el9 appstream 14 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-mono-fonts noarch 1:2.1.3-5.el9 appstream 497 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-sans-fonts noarch 1:2.1.3-5.el9 appstream 601 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-serif-fonts noarch 1:2.1.3-5.el9 appstream 601 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libexif x86_64 0.6.22-6.el9 appstream 435 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libgexiv2 x86_64 0.12.3-1.el9 appstream 83 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd x86_64 1:1.3.4-1.el9 appstream 135 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-egl x86_64 1:1.3.4-1.el9 appstream 37 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-glx x86_64 1:1.3.4-1.el9 appstream 142 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libgsf x86_64 1.14.47-5.el9 appstream 256 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libgxps x86_64 0.3.2-3.el9 appstream 79 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libiptcdata x86_64 1.0.5-9.el9 appstream 62 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libldac x86_64 2.0.2.3-10.el9 appstream 41 k 2024-04-26T04:22:58.416 INFO:tasks.workunit.client.0.smithi098.stdout: libnotify x86_64 0.7.9-8.el9 appstream 44 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libogg x86_64 2:1.3.4-6.el9 appstream 34 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libosinfo x86_64 1.10.0-1.el9 appstream 319 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libpciaccess x86_64 0.16-7.el9 baseos 27 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libsbc x86_64 1.4-9.el9 appstream 45 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libsndfile x86_64 1.0.31-8.el9 appstream 206 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libtheora x86_64 1:1.1.1-31.el9 appstream 164 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libtiff x86_64 4.4.0-12.el9 appstream 198 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libtool-ltdl x86_64 2.4.6-45.el9 appstream 37 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libtracker-sparql x86_64 3.1.2-3.el9 appstream 324 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libvisual x86_64 1:0.4.0-34.el9 appstream 144 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libvorbis x86_64 1:1.3.7-5.el9 appstream 193 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-client x86_64 1.21.0-1.el9 appstream 33 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-cursor x86_64 1.21.0-1.el9 appstream 19 k 2024-04-26T04:22:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-egl x86_64 1.21.0-1.el9 appstream 13 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-server x86_64 1.21.0-1.el9 appstream 42 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: libwebp x86_64 1.2.0-8.el9 appstream 277 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbcommon x86_64 1.0.3-4.el9 appstream 133 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: libxshmfence x86_64 1.3-10.el9 appstream 13 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: low-memory-monitor x86_64 2.1-4.el9 appstream 36 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libEGL x86_64 23.3.3-1.el9 appstream 127 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libGL x86_64 23.3.3-1.el9 appstream 168 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libgbm x86_64 23.3.3-1.el9 appstream 38 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libglapi x86_64 23.3.3-1.el9 appstream 44 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-vulkan-drivers x86_64 23.3.3-1.el9 appstream 9.7 M 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: openjpeg2 x86_64 2.4.0-7.el9 appstream 163 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: opus x86_64 1.3.1-10.el9 appstream 200 k 2024-04-26T04:22:58.418 INFO:tasks.workunit.client.0.smithi098.stdout: orc x86_64 0.4.31-6.el9 appstream 184 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: osinfo-db noarch 20231215-1.el9 appstream 521 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: osinfo-db-tools x86_64 1.10.0-1.el9 appstream 74 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: ostree-libs x86_64 2024.5-2.el9 appstream 488 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-jack-audio-connection-kit-libs x86_64 1.0.1-1.el9 appstream 135 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-libs x86_64 1.0.1-1.el9 appstream 1.9 M 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: poppler x86_64 21.01.0-19.el9 appstream 1.1 M 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: poppler-data noarch 0.4.9-9.el9 appstream 1.9 M 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: poppler-glib x86_64 21.01.0-19.el9 appstream 152 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: pulseaudio-libs x86_64 15.0-2.el9 appstream 679 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: rtkit x86_64 0.11-28.el9 appstream 56 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: sound-theme-freedesktop noarch 0.8-17.el9 appstream 383 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: totem-pl-parser x86_64 3.26.6-2.el9 appstream 152 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: tracker x86_64 3.1.2-3.el9 appstream 555 k 2024-04-26T04:22:58.419 INFO:tasks.workunit.client.0.smithi098.stdout: upower x86_64 0.99.11-11.el9 appstream 173 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: vulkan-loader x86_64 1.3.268.0-1.el9 appstream 145 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: webrtc-audio-processing x86_64 0.3.1-8.el9 appstream 306 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: wireplumber x86_64 0.4.14-1.el9 appstream 96 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: wireplumber-libs x86_64 0.4.14-1.el9 appstream 358 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-dbus-proxy x86_64 0.1.3-1.el9 appstream 42 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-desktop-portal x86_64 1.12.6-1.el9 appstream 385 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: xkeyboard-config noarch 2.33-2.el9 appstream 859 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout:Installing weak dependencies: 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: dconf x86_64 0.40.0-6.el9 appstream 113 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: exiv2 x86_64 0.27.5-2.el9 appstream 980 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak x86_64 1.12.8-1.el9 appstream 1.7 M 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra-gtk3 x86_64 0.30-27.el9 appstream 32 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: p11-kit-server x86_64 0.25.3-2.el9 appstream 246 k 2024-04-26T04:22:58.420 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire x86_64 1.0.1-1.el9 appstream 107 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-alsa x86_64 1.0.1-1.el9 appstream 57 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-jack-audio-connection-kit x86_64 1.0.1-1.el9 appstream 9.1 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-pulseaudio x86_64 1.0.1-1.el9 appstream 192 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: tracker-miners x86_64 3.1.2-4.el9 appstream 944 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-desktop-portal-gtk x86_64 1.12.0-3.el9 appstream 134 k 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction Summary 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout:======================================================================================================== 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout:Install 121 Packages 2024-04-26T04:22:58.421 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:22:58.422 INFO:tasks.workunit.client.0.smithi098.stdout:Total download size: 165 M 2024-04-26T04:22:58.422 INFO:tasks.workunit.client.0.smithi098.stdout:Installed size: 545 M 2024-04-26T04:22:58.422 INFO:tasks.workunit.client.0.smithi098.stdout:Downloading Packages: 2024-04-26T04:22:58.942 INFO:tasks.workunit.client.0.smithi098.stdout:(1/121): bubblewrap-0.4.1-7.el9.x86_64.rpm 303 kB/s | 49 kB 00:00 2024-04-26T04:22:58.968 INFO:tasks.workunit.client.0.smithi098.stdout:(2/121): bluez-libs-5.56-6.el9.x86_64.rpm 449 kB/s | 85 kB 00:00 2024-04-26T04:22:59.018 INFO:tasks.workunit.client.0.smithi098.stdout:(3/121): ModemManager-glib-1.20.2-1.el9.x86_64. 1.4 MB/s | 335 kB 00:00 2024-04-26T04:22:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:22:58 smithi098 ceph-mon[27184]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:22:59.052 INFO:tasks.workunit.client.0.smithi098.stdout:(4/121): cups-libs-2.3.3op2-24.el9.x86_64.rpm 2.4 MB/s | 262 kB 00:00 2024-04-26T04:22:59.077 INFO:tasks.workunit.client.0.smithi098.stdout:(5/121): libpciaccess-0.16-7.el9.x86_64.rpm 247 kB/s | 27 kB 00:00 2024-04-26T04:22:59.127 INFO:tasks.workunit.client.0.smithi098.stdout:(6/121): adwaita-cursor-theme-40.1.1-3.el9.noar 5.8 MB/s | 650 kB 00:00 2024-04-26T04:22:59.219 INFO:tasks.workunit.client.0.smithi098.stdout:(7/121): alsa-lib-1.2.10-2.el9.x86_64.rpm 3.6 MB/s | 523 kB 00:00 2024-04-26T04:22:59.261 INFO:tasks.workunit.client.0.smithi098.stdout:(8/121): at-spi2-atk-2.38.0-4.el9.x86_64.rpm 657 kB/s | 87 kB 00:00 2024-04-26T04:22:59.504 INFO:tasks.workunit.client.0.smithi098.stdout:(9/121): adwaita-icon-theme-40.1.1-3.el9.noarch 26 MB/s | 12 MB 00:00 2024-04-26T04:22:59.529 INFO:tasks.workunit.client.0.smithi098.stdout:(10/121): at-spi2-core-2.40.3-1.el9.x86_64.rpm 635 kB/s | 195 kB 00:00 2024-04-26T04:22:59.554 INFO:tasks.workunit.client.0.smithi098.stdout:(11/121): atk-2.36.0-5.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-04-26T04:22:59.579 INFO:tasks.workunit.client.0.smithi098.stdout:(12/121): avahi-glib-0.8-20.el9.x86_64.rpm 201 kB/s | 15 kB 00:00 2024-04-26T04:22:59.613 INFO:tasks.workunit.client.0.smithi098.stdout:(13/121): dconf-0.40.0-6.el9.x86_64.rpm 1.9 MB/s | 113 kB 00:00 2024-04-26T04:22:59.639 INFO:tasks.workunit.client.0.smithi098.stdout:(14/121): colord-libs-1.4.5-4.el9.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2024-04-26T04:22:59.672 INFO:tasks.workunit.client.0.smithi098.stdout:(15/121): exempi-2.6.0-0.2.20211007gite23c213.e 5.6 MB/s | 525 kB 00:00 2024-04-26T04:22:59.706 INFO:tasks.workunit.client.0.smithi098.stdout:(16/121): exiv2-0.27.5-2.el9.x86_64.rpm 10 MB/s | 980 kB 00:00 2024-04-26T04:22:59.731 INFO:tasks.workunit.client.0.smithi098.stdout:(17/121): fdk-aac-free-2.0.0-8.el9.x86_64.rpm 5.6 MB/s | 325 kB 00:00 2024-04-26T04:22:59.756 INFO:tasks.workunit.client.0.smithi098.stdout:(18/121): exiv2-libs-0.27.5-2.el9.x86_64.rpm 9.2 MB/s | 781 kB 00:00 2024-04-26T04:22:59.782 INFO:tasks.workunit.client.0.smithi098.stdout:(19/121): flac-libs-1.3.3-12.el9.x86_64.rpm 2.9 MB/s | 218 kB 00:00 2024-04-26T04:22:59.835 INFO:tasks.workunit.client.0.smithi098.stdout:(20/121): flatpak-1.12.8-1.el9.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2024-04-26T04:22:59.858 INFO:tasks.workunit.client.0.smithi098.stdout:(21/121): flatpak-selinux-1.12.8-1.el9.noarch.r 294 kB/s | 21 kB 00:00 2024-04-26T04:22:59.882 INFO:tasks.workunit.client.0.smithi098.stdout:(22/121): flatpak-session-helper-1.12.8-1.el9.x 755 kB/s | 74 kB 00:00 2024-04-26T04:22:59.909 INFO:tasks.workunit.client.0.smithi098.stdout:(23/121): gdk-pixbuf2-modules-2.42.6-3.el9.x86_ 1.1 MB/s | 88 kB 00:00 2024-04-26T04:22:59.933 INFO:tasks.workunit.client.0.smithi098.stdout:(24/121): giflib-5.2.1-9.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-04-26T04:22:59.958 INFO:tasks.workunit.client.0.smithi098.stdout:(25/121): geoclue2-2.6.0-7.el9.x86_64.rpm 1.6 MB/s | 124 kB 00:00 2024-04-26T04:22:59.983 INFO:tasks.workunit.client.0.smithi098.stdout:(26/121): graphene-1.10.6-2.el9.x86_64.rpm 856 kB/s | 64 kB 00:00 2024-04-26T04:23:00.009 INFO:tasks.workunit.client.0.smithi098.stdout:(27/121): gsm-1.0.19-6.el9.x86_64.rpm 452 kB/s | 34 kB 00:00 2024-04-26T04:23:00.076 INFO:tasks.workunit.client.0.smithi098.stdout:(28/121): gstreamer1-1.22.1-2.el9.x86_64.rpm 16 MB/s | 1.4 MB 00:00 2024-04-26T04:23:00.101 INFO:tasks.workunit.client.0.smithi098.stdout:(29/121): gtk-update-icon-cache-3.24.31-2.el9.x 387 kB/s | 35 kB 00:00 2024-04-26T04:23:00.160 INFO:tasks.workunit.client.0.smithi098.stdout:(30/121): gstreamer1-plugins-base-1.22.1-2.el9. 13 MB/s | 2.2 MB 00:00 2024-04-26T04:23:00.185 INFO:tasks.workunit.client.0.smithi098.stdout:(31/121): hicolor-icon-theme-0.17-13.el9.noarch 2.6 MB/s | 219 kB 00:00 2024-04-26T04:23:00.252 INFO:tasks.workunit.client.0.smithi098.stdout:(32/121): gtk3-3.24.31-2.el9.x86_64.rpm 28 MB/s | 4.9 MB 00:00 2024-04-26T04:23:00.277 INFO:tasks.workunit.client.0.smithi098.stdout:(33/121): jbigkit-libs-2.1-23.el9.x86_64.rpm 594 kB/s | 53 kB 00:00 2024-04-26T04:23:00.344 INFO:tasks.workunit.client.0.smithi098.stdout:(34/121): iso-codes-4.6.0-3.el9.noarch.rpm 19 MB/s | 3.5 MB 00:00 2024-04-26T04:23:00.369 INFO:tasks.workunit.client.0.smithi098.stdout:(35/121): lcms2-2.12-3.el9.x86_64.rpm 1.4 MB/s | 168 kB 00:00 2024-04-26T04:23:00.395 INFO:tasks.workunit.client.0.smithi098.stdout:(36/121): libX11-xcb-1.7.0-9.el9.x86_64.rpm 96 kB/s | 11 kB 00:00 2024-04-26T04:23:00.420 INFO:tasks.workunit.client.0.smithi098.stdout:(37/121): libXcomposite-0.4.5-7.el9.x86_64.rpm 481 kB/s | 24 kB 00:00 2024-04-26T04:23:00.446 INFO:tasks.workunit.client.0.smithi098.stdout:(38/121): libXcursor-1.2.0-7.el9.x86_64.rpm 408 kB/s | 31 kB 00:00 2024-04-26T04:23:00.470 INFO:tasks.workunit.client.0.smithi098.stdout:(39/121): libXdamage-1.1.5-7.el9.x86_64.rpm 303 kB/s | 23 kB 00:00 2024-04-26T04:23:00.504 INFO:tasks.workunit.client.0.smithi098.stdout:(40/121): libXfixes-5.0.3-16.el9.x86_64.rpm 336 kB/s | 20 kB 00:00 2024-04-26T04:23:00.548 INFO:tasks.workunit.client.0.smithi098.stdout:(41/121): libXi-1.7.10-8.el9.x86_64.rpm 386 kB/s | 40 kB 00:00 2024-04-26T04:23:00.574 INFO:tasks.workunit.client.0.smithi098.stdout:(42/121): libXinerama-1.1.4-10.el9.x86_64.rpm 147 kB/s | 15 kB 00:00 2024-04-26T04:23:00.599 INFO:tasks.workunit.client.0.smithi098.stdout:(43/121): libXtst-1.2.3-16.el9.x86_64.rpm 429 kB/s | 21 kB 00:00 2024-04-26T04:23:00.624 INFO:tasks.workunit.client.0.smithi098.stdout:(44/121): libXrandr-1.5.2-8.el9.x86_64.rpm 377 kB/s | 28 kB 00:00 2024-04-26T04:23:00.649 INFO:tasks.workunit.client.0.smithi098.stdout:(45/121): libXv-1.0.11-16.el9.x86_64.rpm 253 kB/s | 19 kB 00:00 2024-04-26T04:23:00.675 INFO:tasks.workunit.client.0.smithi098.stdout:(46/121): libXxf86vm-1.1.4-18.el9.x86_64.rpm 250 kB/s | 19 kB 00:00 2024-04-26T04:23:00.700 INFO:tasks.workunit.client.0.smithi098.stdout:(47/121): libasyncns-0.8-22.el9.x86_64.rpm 608 kB/s | 30 kB 00:00 2024-04-26T04:23:00.725 INFO:tasks.workunit.client.0.smithi098.stdout:(48/121): libcanberra-0.30-27.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2024-04-26T04:23:00.751 INFO:tasks.workunit.client.0.smithi098.stdout:(49/121): libcanberra-gtk3-0.30-27.el9.x86_64.r 420 kB/s | 32 kB 00:00 2024-04-26T04:23:00.776 INFO:tasks.workunit.client.0.smithi098.stdout:(50/121): libdrm-2.4.117-1.el9.x86_64.rpm 3.1 MB/s | 161 kB 00:00 2024-04-26T04:23:00.801 INFO:tasks.workunit.client.0.smithi098.stdout:(51/121): libepoxy-1.5.5-4.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-26T04:23:00.827 INFO:tasks.workunit.client.0.smithi098.stdout:(52/121): liberation-fonts-2.1.3-5.el9.noarch.r 48 kB/s | 3.6 kB 00:00 2024-04-26T04:23:00.863 INFO:tasks.workunit.client.0.smithi098.stdout:(53/121): liberation-fonts-common-2.1.3-5.el9.n 228 kB/s | 14 kB 00:00 2024-04-26T04:23:00.894 INFO:tasks.workunit.client.0.smithi098.stdout:(54/121): liberation-mono-fonts-2.1.3-5.el9.noa 5.3 MB/s | 497 kB 00:00 2024-04-26T04:23:00.920 INFO:tasks.workunit.client.0.smithi098.stdout:(55/121): liberation-sans-fonts-2.1.3-5.el9.noa 6.3 MB/s | 601 kB 00:00 2024-04-26T04:23:00.953 INFO:tasks.workunit.client.0.smithi098.stdout:(56/121): libexif-0.6.22-6.el9.x86_64.rpm 7.5 MB/s | 435 kB 00:00 2024-04-26T04:23:00.987 INFO:tasks.workunit.client.0.smithi098.stdout:(57/121): liberation-serif-fonts-2.1.3-5.el9.no 6.5 MB/s | 601 kB 00:00 2024-04-26T04:23:01.012 INFO:tasks.workunit.client.0.smithi098.stdout:(58/121): libgexiv2-0.12.3-1.el9.x86_64.rpm 907 kB/s | 83 kB 00:00 2024-04-26T04:23:01.037 INFO:tasks.workunit.client.0.smithi098.stdout:(59/121): libglvnd-1.3.4-1.el9.x86_64.rpm 1.6 MB/s | 135 kB 00:00 2024-04-26T04:23:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:00 smithi098 ceph-mon[27184]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:00 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:23:01.062 INFO:tasks.workunit.client.0.smithi098.stdout:(60/121): libglvnd-egl-1.3.4-1.el9.x86_64.rpm 744 kB/s | 37 kB 00:00 2024-04-26T04:23:01.105 INFO:tasks.workunit.client.0.smithi098.stdout:(61/121): libglvnd-glx-1.3.4-1.el9.x86_64.rpm 1.5 MB/s | 142 kB 00:00 2024-04-26T04:23:01.163 INFO:tasks.workunit.client.0.smithi098.stdout:(62/121): libgsf-1.14.47-5.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-04-26T04:23:01.189 INFO:tasks.workunit.client.0.smithi098.stdout:(63/121): libgxps-0.3.2-3.el9.x86_64.rpm 627 kB/s | 79 kB 00:00 2024-04-26T04:23:01.214 INFO:tasks.workunit.client.0.smithi098.stdout:(64/121): libiptcdata-1.0.5-9.el9.x86_64.rpm 571 kB/s | 62 kB 00:00 2024-04-26T04:23:01.239 INFO:tasks.workunit.client.0.smithi098.stdout:(65/121): libldac-2.0.2.3-10.el9.x86_64.rpm 548 kB/s | 41 kB 00:00 2024-04-26T04:23:01.264 INFO:tasks.workunit.client.0.smithi098.stdout:(66/121): libnotify-0.7.9-8.el9.x86_64.rpm 578 kB/s | 44 kB 00:00 2024-04-26T04:23:01.299 INFO:tasks.workunit.client.0.smithi098.stdout:(67/121): libogg-1.3.4-6.el9.x86_64.rpm 569 kB/s | 34 kB 00:00 2024-04-26T04:23:01.382 INFO:tasks.workunit.client.0.smithi098.stdout:(68/121): libosinfo-1.10.0-1.el9.x86_64.rpm 2.2 MB/s | 319 kB 00:00 2024-04-26T04:23:01.407 INFO:tasks.workunit.client.0.smithi098.stdout:(69/121): libsbc-1.4-9.el9.x86_64.rpm 314 kB/s | 45 kB 00:00 2024-04-26T04:23:01.433 INFO:tasks.workunit.client.0.smithi098.stdout:(70/121): libsndfile-1.0.31-8.el9.x86_64.rpm 1.5 MB/s | 206 kB 00:00 2024-04-26T04:23:01.475 INFO:tasks.workunit.client.0.smithi098.stdout:(71/121): libtheora-1.1.1-31.el9.x86_64.rpm 1.7 MB/s | 164 kB 00:00 2024-04-26T04:23:01.508 INFO:tasks.workunit.client.0.smithi098.stdout:(72/121): libtiff-4.4.0-12.el9.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2024-04-26T04:23:01.534 INFO:tasks.workunit.client.0.smithi098.stdout:(73/121): libtool-ltdl-2.4.6-45.el9.x86_64.rpm 368 kB/s | 37 kB 00:00 2024-04-26T04:23:01.609 INFO:tasks.workunit.client.0.smithi098.stdout:(74/121): libtracker-sparql-3.1.2-3.el9.x86_64. 2.4 MB/s | 324 kB 00:00 2024-04-26T04:23:01.693 INFO:tasks.workunit.client.0.smithi098.stdout:(75/121): libvisual-0.4.0-34.el9.x86_64.rpm 782 kB/s | 144 kB 00:00 2024-04-26T04:23:01.718 INFO:tasks.workunit.client.0.smithi098.stdout:(76/121): libvorbis-1.3.7-5.el9.x86_64.rpm 1.0 MB/s | 193 kB 00:00 2024-04-26T04:23:01.735 INFO:tasks.workunit.client.0.smithi098.stdout:(77/121): libwayland-client-1.21.0-1.el9.x86_64 266 kB/s | 33 kB 00:00 2024-04-26T04:23:01.768 INFO:tasks.workunit.client.0.smithi098.stdout:(78/121): libwayland-egl-1.21.0-1.el9.x86_64.rp 257 kB/s | 13 kB 00:00 2024-04-26T04:23:01.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:23:01.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:23:01.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:23:01.794 INFO:tasks.workunit.client.0.smithi098.stdout:(79/121): libwayland-cursor-1.21.0-1.el9.x86_64 257 kB/s | 19 kB 00:00 2024-04-26T04:23:01.819 INFO:tasks.workunit.client.0.smithi098.stdout:(80/121): libwayland-server-1.21.0-1.el9.x86_64 496 kB/s | 42 kB 00:00 2024-04-26T04:23:01.903 INFO:tasks.workunit.client.0.smithi098.stdout:(81/121): libwebp-1.2.0-8.el9.x86_64.rpm 2.0 MB/s | 277 kB 00:00 2024-04-26T04:23:01.928 INFO:tasks.workunit.client.0.smithi098.stdout:(82/121): libxshmfence-1.3-10.el9.x86_64.rpm 116 kB/s | 13 kB 00:00 2024-04-26T04:23:01.953 INFO:tasks.workunit.client.0.smithi098.stdout:(83/121): libxkbcommon-1.0.3-4.el9.x86_64.rpm 998 kB/s | 133 kB 00:00 2024-04-26T04:23:01.979 INFO:tasks.workunit.client.0.smithi098.stdout:(84/121): low-memory-monitor-2.1-4.el9.x86_64.r 475 kB/s | 36 kB 00:00 2024-04-26T04:23:02.012 INFO:tasks.workunit.client.0.smithi098.stdout:(85/121): mesa-libEGL-23.3.3-1.el9.x86_64.rpm 1.5 MB/s | 127 kB 00:00 2024-04-26T04:23:02.046 INFO:tasks.workunit.client.0.smithi098.stdout:(86/121): mesa-libGL-23.3.3-1.el9.x86_64.rpm 1.8 MB/s | 168 kB 00:00 2024-04-26T04:23:02.071 INFO:tasks.workunit.client.0.smithi098.stdout:(87/121): mesa-libgbm-23.3.3-1.el9.x86_64.rpm 407 kB/s | 38 kB 00:00 2024-04-26T04:23:02.107 INFO:tasks.workunit.client.0.smithi098.stdout:(88/121): mesa-libglapi-23.3.3-1.el9.x86_64.rpm 470 kB/s | 44 kB 00:00 2024-04-26T04:23:02.464 INFO:tasks.workunit.client.0.smithi098.stdout:(89/121): opus-1.3.1-10.el9.x86_64.rpm 562 kB/s | 200 kB 00:00 2024-04-26T04:23:02.489 INFO:tasks.workunit.client.0.smithi098.stdout:(90/121): openjpeg2-2.4.0-7.el9.x86_64.rpm 428 kB/s | 163 kB 00:00 2024-04-26T04:23:02.673 INFO:tasks.workunit.client.0.smithi098.stdout:(91/121): mesa-vulkan-drivers-23.3.3-1.el9.x86_ 15 MB/s | 9.7 MB 00:00 2024-04-26T04:23:02.698 INFO:tasks.workunit.client.0.smithi098.stdout:(92/121): orc-0.4.31-6.el9.x86_64.rpm 787 kB/s | 184 kB 00:00 2024-04-26T04:23:02.726 INFO:tasks.workunit.client.0.smithi098.stdout:(93/121): osinfo-db-20231215-1.el9.noarch.rpm 2.2 MB/s | 521 kB 00:00 2024-04-26T04:23:02.752 INFO:tasks.workunit.client.0.smithi098.stdout:(94/121): osinfo-db-tools-1.10.0-1.el9.x86_64.r 945 kB/s | 74 kB 00:00 2024-04-26T04:23:02.794 INFO:tasks.workunit.client.0.smithi098.stdout:(95/121): p11-kit-server-0.25.3-2.el9.x86_64.rp 3.6 MB/s | 246 kB 00:00 2024-04-26T04:23:02.828 INFO:tasks.workunit.client.0.smithi098.stdout:(96/121): ostree-libs-2024.5-2.el9.x86_64.rpm 4.7 MB/s | 488 kB 00:00 2024-04-26T04:23:02.853 INFO:tasks.workunit.client.0.smithi098.stdout:(97/121): pipewire-1.0.1-1.el9.x86_64.rpm 1.0 MB/s | 107 kB 00:00 2024-04-26T04:23:02.878 INFO:tasks.workunit.client.0.smithi098.stdout:(98/121): pipewire-alsa-1.0.1-1.el9.x86_64.rpm 691 kB/s | 57 kB 00:00 2024-04-26T04:23:02.912 INFO:tasks.workunit.client.0.smithi098.stdout:(99/121): pipewire-jack-audio-connection-kit-1. 157 kB/s | 9.1 kB 00:00 2024-04-26T04:23:02.937 INFO:tasks.workunit.client.0.smithi098.stdout:(100/121): pipewire-jack-audio-connection-kit-l 1.6 MB/s | 135 kB 00:00 2024-04-26T04:23:02.979 INFO:tasks.workunit.client.0.smithi098.stdout:(101/121): pipewire-libs-1.0.1-1.el9.x86_64.rpm 19 MB/s | 1.9 MB 00:00 2024-04-26T04:23:03.004 INFO:tasks.workunit.client.0.smithi098.stdout:(102/121): pipewire-pulseaudio-1.0.1-1.el9.x86_ 2.1 MB/s | 192 kB 00:00 2024-04-26T04:23:03.038 INFO:tasks.workunit.client.0.smithi098.stdout:(103/121): poppler-21.01.0-19.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-04-26T04:23:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:02 smithi098 ceph-mon[27184]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:03.063 INFO:tasks.workunit.client.0.smithi098.stdout:(104/121): poppler-glib-21.01.0-19.el9.x86_64.r 2.6 MB/s | 152 kB 00:00 2024-04-26T04:23:03.105 INFO:tasks.workunit.client.0.smithi098.stdout:(105/121): poppler-data-0.4.9-9.el9.noarch.rpm 19 MB/s | 1.9 MB 00:00 2024-04-26T04:23:03.130 INFO:tasks.workunit.client.0.smithi098.stdout:(106/121): pulseaudio-libs-15.0-2.el9.x86_64.rp 7.2 MB/s | 679 kB 00:00 2024-04-26T04:23:03.156 INFO:tasks.workunit.client.0.smithi098.stdout:(107/121): rtkit-0.11-28.el9.x86_64.rpm 622 kB/s | 56 kB 00:00 2024-04-26T04:23:03.181 INFO:tasks.workunit.client.0.smithi098.stdout:(108/121): sound-theme-freedesktop-0.8-17.el9.n 5.0 MB/s | 383 kB 00:00 2024-04-26T04:23:03.206 INFO:tasks.workunit.client.0.smithi098.stdout:(109/121): totem-pl-parser-3.26.6-2.el9.x86_64. 3.0 MB/s | 152 kB 00:00 2024-04-26T04:23:03.240 INFO:tasks.workunit.client.0.smithi098.stdout:(110/121): tracker-3.1.2-3.el9.x86_64.rpm 6.5 MB/s | 555 kB 00:00 2024-04-26T04:23:03.274 INFO:tasks.workunit.client.0.smithi098.stdout:(111/121): tracker-miners-3.1.2-4.el9.x86_64.rp 10 MB/s | 944 kB 00:00 2024-04-26T04:23:03.299 INFO:tasks.workunit.client.0.smithi098.stdout:(112/121): upower-0.99.11-11.el9.x86_64.rpm 1.9 MB/s | 173 kB 00:00 2024-04-26T04:23:03.324 INFO:tasks.workunit.client.0.smithi098.stdout:(113/121): vulkan-loader-1.3.268.0-1.el9.x86_64 1.7 MB/s | 145 kB 00:00 2024-04-26T04:23:03.349 INFO:tasks.workunit.client.0.smithi098.stdout:(114/121): wireplumber-0.4.14-1.el9.x86_64.rpm 1.9 MB/s | 96 kB 00:00 2024-04-26T04:23:03.383 INFO:tasks.workunit.client.0.smithi098.stdout:(115/121): webrtc-audio-processing-0.3.1-8.el9. 3.6 MB/s | 306 kB 00:00 2024-04-26T04:23:03.408 INFO:tasks.workunit.client.0.smithi098.stdout:(116/121): wireplumber-libs-0.4.14-1.el9.x86_64 4.2 MB/s | 358 kB 00:00 2024-04-26T04:23:03.434 INFO:tasks.workunit.client.0.smithi098.stdout:(117/121): xdg-dbus-proxy-0.1.3-1.el9.x86_64.rp 510 kB/s | 42 kB 00:00 2024-04-26T04:23:03.459 INFO:tasks.workunit.client.0.smithi098.stdout:(118/121): xdg-desktop-portal-gtk-1.12.0-3.el9. 2.6 MB/s | 134 kB 00:00 2024-04-26T04:23:03.484 INFO:tasks.workunit.client.0.smithi098.stdout:(119/121): xdg-desktop-portal-1.12.6-1.el9.x86_ 5.0 MB/s | 385 kB 00:00 2024-04-26T04:23:03.509 INFO:tasks.workunit.client.0.smithi098.stdout:(120/121): xkeyboard-config-2.33-2.el9.noarch.r 11 MB/s | 859 kB 00:00 2024-04-26T04:23:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:04 smithi098 ceph-mon[27184]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:06.210 INFO:tasks.workunit.client.0.smithi098.stdout:(121/121): google-chrome-stable-124.0.6367.91-1 38 MB/s | 104 MB 00:02 2024-04-26T04:23:06.215 INFO:tasks.workunit.client.0.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:23:06.215 INFO:tasks.workunit.client.0.smithi098.stdout:Total 21 MB/s | 165 MB 00:07 2024-04-26T04:23:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:06 smithi098 ceph-mon[27184]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:08.707 INFO:tasks.workunit.client.0.smithi098.stdout:google-chrome 132 kB/s | 16 kB 00:00 2024-04-26T04:23:08.821 INFO:tasks.workunit.client.0.smithi098.stderr:Importing GPG key 0x7FAC5991: 2024-04-26T04:23:08.821 INFO:tasks.workunit.client.0.smithi098.stderr: Userid : "Google, Inc. Linux Package Signing Key " 2024-04-26T04:23:08.821 INFO:tasks.workunit.client.0.smithi098.stderr: Fingerprint: 4CCA 1EAF 950C EE4A B839 76DC A040 830F 7FAC 5991 2024-04-26T04:23:08.821 INFO:tasks.workunit.client.0.smithi098.stderr: From : https://dl-ssl.google.com/linux/linux_signing_key.pub 2024-04-26T04:23:08.877 INFO:tasks.workunit.client.0.smithi098.stdout:Key imported successfully 2024-04-26T04:23:08.878 INFO:tasks.workunit.client.0.smithi098.stderr:Importing GPG key 0xD38B4796: 2024-04-26T04:23:08.878 INFO:tasks.workunit.client.0.smithi098.stderr: Userid : "Google Inc. (Linux Packages Signing Authority) " 2024-04-26T04:23:08.878 INFO:tasks.workunit.client.0.smithi098.stderr: Fingerprint: EB4C 1BFD 4F04 2F6D DDCC EC91 7721 F63B D38B 4796 2024-04-26T04:23:08.878 INFO:tasks.workunit.client.0.smithi098.stderr: From : https://dl-ssl.google.com/linux/linux_signing_key.pub 2024-04-26T04:23:08.902 INFO:tasks.workunit.client.0.smithi098.stdout:Key imported successfully 2024-04-26T04:23:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:08 smithi098 ceph-mon[27184]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:09.680 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction check 2024-04-26T04:23:09.768 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:23:09.768 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction test 2024-04-26T04:23:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:10 smithi098 ceph-mon[27184]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:11.340 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:23:11.340 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction 2024-04-26T04:23:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:12 smithi098 ceph-mon[27184]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:14 smithi098 ceph-mon[27184]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:16.011 INFO:tasks.workunit.client.0.smithi098.stdout: Preparing : 1/1 2024-04-26T04:23:16.081 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libwayland-client-1.21.0-1.el9.x86_64 1/121 2024-04-26T04:23:16.106 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libogg-2:1.3.4-6.el9.x86_64 2/121 2024-04-26T04:23:16.174 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXfixes-5.0.3-16.el9.x86_64 3/121 2024-04-26T04:23:16.213 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : alsa-lib-1.2.10-2.el9.x86_64 4/121 2024-04-26T04:23:16.242 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libvorbis-1:1.3.7-5.el9.x86_64 5/121 2024-04-26T04:23:16.266 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXi-1.7.10-8.el9.x86_64 6/121 2024-04-26T04:23:16.341 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libX11-xcb-1.7.0-9.el9.x86_64 7/121 2024-04-26T04:23:16.382 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : atk-2.36.0-5.el9.x86_64 8/121 2024-04-26T04:23:16.415 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : opus-1.3.1-10.el9.x86_64 9/121 2024-04-26T04:23:16.441 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxshmfence-1.3-10.el9.x86_64 10/121 2024-04-26T04:23:16.467 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : liberation-fonts-common-1:2.1.3-5.el9.noarch 11/121 2024-04-26T04:23:16.558 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : lcms2-2.12-3.el9.x86_64 12/121 2024-04-26T04:23:16.604 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gstreamer1-1.22.1-2.el9.x86_64 13/121 2024-04-26T04:23:16.628 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXdamage-1.1.5-7.el9.x86_64 14/121 2024-04-26T04:23:16.653 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libwayland-cursor-1.21.0-1.el9.x86_64 15/121 2024-04-26T04:23:16.679 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : mesa-libglapi-23.3.3-1.el9.x86_64 16/121 2024-04-26T04:23:16.704 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libwayland-server-1.21.0-1.el9.x86_64 17/121 2024-04-26T04:23:16.730 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libwayland-egl-1.21.0-1.el9.x86_64 18/121 2024-04-26T04:23:16.759 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libtool-ltdl-2.4.6-45.el9.x86_64 19/121 2024-04-26T04:23:16.790 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-1:1.3.4-1.el9.x86_64 20/121 2024-04-26T04:23:16.816 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXrandr-1.5.2-8.el9.x86_64 21/121 2024-04-26T04:23:16.848 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXcomposite-0.4.5-7.el9.x86_64 22/121 2024-04-26T04:23:16.881 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : dconf-0.40.0-6.el9.x86_64 23/121 2024-04-26T04:23:16.914 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 23/121 2024-04-26T04:23:16.948 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : cups-libs-1:2.3.3op2-24.el9.x86_64 24/121 2024-04-26T04:23:16.985 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : colord-libs-1.4.5-4.el9.x86_64 25/121 2024-04-26T04:23:17.029 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : liberation-mono-fonts-1:2.1.3-5.el9.noarch 26/121 2024-04-26T04:23:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:16 smithi098 ceph-mon[27184]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:17.062 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : liberation-sans-fonts-1:2.1.3-5.el9.noarch 27/121 2024-04-26T04:23:17.085 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : liberation-serif-fonts-1:2.1.3-5.el9.noarch 28/121 2024-04-26T04:23:17.111 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : liberation-fonts-1:2.1.3-5.el9.noarch 29/121 2024-04-26T04:23:17.174 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXtst-1.2.3-16.el9.x86_64 30/121 2024-04-26T04:23:17.211 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : at-spi2-core-2.40.3-1.el9.x86_64 31/121 2024-04-26T04:23:17.243 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : at-spi2-atk-2.38.0-4.el9.x86_64 32/121 2024-04-26T04:23:17.272 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXcursor-1.2.0-7.el9.x86_64 33/121 2024-04-26T04:23:17.304 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : flac-libs-1.3.3-12.el9.x86_64 34/121 2024-04-26T04:23:17.347 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libtheora-1:1.1.1-31.el9.x86_64 35/121 2024-04-26T04:23:17.413 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : exiv2-libs-0.27.5-2.el9.x86_64 36/121 2024-04-26T04:23:17.448 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : exiv2-0.27.5-2.el9.x86_64 37/121 2024-04-26T04:23:17.651 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libgexiv2-0.12.3-1.el9.x86_64 38/121 2024-04-26T04:23:17.701 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xkeyboard-config-2.33-2.el9.noarch 39/121 2024-04-26T04:23:17.733 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxkbcommon-1.0.3-4.el9.x86_64 40/121 2024-04-26T04:23:17.763 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xdg-dbus-proxy-0.1.3-1.el9.x86_64 41/121 2024-04-26T04:23:17.805 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : webrtc-audio-processing-0.3.1-8.el9.x86_64 42/121 2024-04-26T04:23:17.842 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : upower-0.99.11-11.el9.x86_64 43/121 2024-04-26T04:23:17.855 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: upower-0.99.11-11.el9.x86_64 43/121 2024-04-26T04:23:17.855 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/system/graphical.target.wants/upower.service → /usr/lib/systemd/system/upower.service. 2024-04-26T04:23:17.855 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:17.955 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : totem-pl-parser-3.26.6-2.el9.x86_64 44/121 2024-04-26T04:23:17.980 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : sound-theme-freedesktop-0.8-17.el9.noarch 45/121 2024-04-26T04:23:17.987 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: sound-theme-freedesktop-0.8-17.el9.noarch 45/121 2024-04-26T04:23:18.402 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 46/121 2024-04-26T04:23:18.443 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : rtkit-0.11-28.el9.x86_64 46/121 2024-04-26T04:23:18.456 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 46/121 2024-04-26T04:23:18.456 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/system/graphical.target.wants/rtkit-daemon.service → /usr/lib/systemd/system/rtkit-daemon.service. 2024-04-26T04:23:18.456 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:18.670 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : poppler-data-0.4.9-9.el9.noarch 47/121 2024-04-26T04:23:18.703 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : p11-kit-server-0.25.3-2.el9.x86_64 48/121 2024-04-26T04:23:18.742 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : ostree-libs-2024.5-2.el9.x86_64 49/121 2024-04-26T04:23:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:18 smithi098 ceph-mon[27184]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:19.248 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : osinfo-db-tools-1.10.0-1.el9.x86_64 50/121 2024-04-26T04:23:19.314 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : osinfo-db-20231215-1.el9.noarch 51/121 2024-04-26T04:23:19.351 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libosinfo-1.10.0-1.el9.x86_64 52/121 2024-04-26T04:23:19.384 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : orc-0.4.31-6.el9.x86_64 53/121 2024-04-26T04:23:19.417 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : openjpeg2-2.4.0-7.el9.x86_64 54/121 2024-04-26T04:23:19.446 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : low-memory-monitor-2.1-4.el9.x86_64 55/121 2024-04-26T04:23:19.458 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: low-memory-monitor-2.1-4.el9.x86_64 55/121 2024-04-26T04:23:19.458 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/system/basic.target.wants/low-memory-monitor.service → /usr/lib/systemd/system/low-memory-monitor.service. 2024-04-26T04:23:19.458 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:19.504 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libwebp-1.2.0-8.el9.x86_64 56/121 2024-04-26T04:23:19.534 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libvisual-1:0.4.0-34.el9.x86_64 57/121 2024-04-26T04:23:19.562 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libsbc-1.4-9.el9.x86_64 58/121 2024-04-26T04:23:19.594 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libnotify-0.7.9-8.el9.x86_64 59/121 2024-04-26T04:23:19.624 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libldac-2.0.2.3-10.el9.x86_64 60/121 2024-04-26T04:23:19.680 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libiptcdata-1.0.5-9.el9.x86_64 61/121 2024-04-26T04:23:19.736 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libgsf-1.14.47-5.el9.x86_64 62/121 2024-04-26T04:23:19.771 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libexif-0.6.22-6.el9.x86_64 63/121 2024-04-26T04:23:19.808 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libepoxy-1.5.5-4.el9.x86_64 64/121 2024-04-26T04:23:19.833 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libasyncns-0.8-22.el9.x86_64 65/121 2024-04-26T04:23:19.859 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXxf86vm-1.1.4-18.el9.x86_64 66/121 2024-04-26T04:23:19.884 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXv-1.0.11-16.el9.x86_64 67/121 2024-04-26T04:23:19.912 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXinerama-1.1.4-10.el9.x86_64 68/121 2024-04-26T04:23:19.948 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : jbigkit-libs-2.1-23.el9.x86_64 69/121 2024-04-26T04:23:19.985 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libtiff-4.4.0-12.el9.x86_64 70/121 2024-04-26T04:23:20.016 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 71/121 2024-04-26T04:23:20.064 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libgxps-0.3.2-3.el9.x86_64 72/121 2024-04-26T04:23:20.099 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : poppler-21.01.0-19.el9.x86_64 73/121 2024-04-26T04:23:20.524 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : poppler-glib-21.01.0-19.el9.x86_64 74/121 2024-04-26T04:23:20.662 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : iso-codes-4.6.0-3.el9.noarch 75/121 2024-04-26T04:23:20.733 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : hicolor-icon-theme-0.17-13.el9.noarch 76/121 2024-04-26T04:23:20.768 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gtk-update-icon-cache-3.24.31-2.el9.x86_64 77/121 2024-04-26T04:23:20.812 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gsm-1.0.19-6.el9.x86_64 78/121 2024-04-26T04:23:20.880 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libsndfile-1.0.31-8.el9.x86_64 79/121 2024-04-26T04:23:20.927 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pulseaudio-libs-15.0-2.el9.x86_64 80/121 2024-04-26T04:23:20.952 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libcanberra-0.30-27.el9.x86_64 81/121 2024-04-26T04:23:20.972 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: libcanberra-0.30-27.el9.x86_64 81/121 2024-04-26T04:23:20.998 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : graphene-1.10.6-2.el9.x86_64 82/121 2024-04-26T04:23:21.025 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : giflib-5.2.1-9.el9.x86_64 83/121 2024-04-26T04:23:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:20 smithi098 ceph-mon[27184]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:21.056 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : flatpak-session-helper-1.12.8-1.el9.x86_64 84/121 2024-04-26T04:23:21.079 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : flatpak-selinux-1.12.8-1.el9.noarch 85/121 2024-04-26T04:23:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:22 smithi098 ceph-mon[27184]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:24 smithi098 ceph-mon[27184]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:26 smithi098 ceph-mon[27184]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:28 smithi098 ceph-mon[27184]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:30.107 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: flatpak-selinux-1.12.8-1.el9.noarch 85/121 2024-04-26T04:23:30.145 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : fdk-aac-free-2.0.0-8.el9.x86_64 86/121 2024-04-26T04:23:30.178 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 87/121 2024-04-26T04:23:30.277 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : avahi-glib-0.8-20.el9.x86_64 88/121 2024-04-26T04:23:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:30 smithi098 ceph-mon[27184]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:32 smithi098 ceph-mon[27184]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:33.124 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : adwaita-cursor-theme-40.1.1-3.el9.noarch 89/121 2024-04-26T04:23:33.236 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : adwaita-icon-theme-40.1.1-3.el9.noarch 90/121 2024-04-26T04:23:33.265 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libpciaccess-0.16-7.el9.x86_64 91/121 2024-04-26T04:23:33.294 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libdrm-2.4.117-1.el9.x86_64 92/121 2024-04-26T04:23:33.320 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : mesa-libgbm-23.3.3-1.el9.x86_64 93/121 2024-04-26T04:23:33.347 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-egl-1:1.3.4-1.el9.x86_64 94/121 2024-04-26T04:23:33.374 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : mesa-libEGL-23.3.3-1.el9.x86_64 95/121 2024-04-26T04:23:33.407 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-glx-1:1.3.4-1.el9.x86_64 96/121 2024-04-26T04:23:33.514 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : mesa-libGL-23.3.3-1.el9.x86_64 97/121 2024-04-26T04:23:33.580 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gstreamer1-plugins-base-1.22.1-2.el9.x86_64 98/121 2024-04-26T04:23:33.649 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libtracker-sparql-3.1.2-3.el9.x86_64 99/121 2024-04-26T04:23:33.686 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : tracker-3.1.2-3.el9.x86_64 100/121 2024-04-26T04:23:33.796 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: tracker-3.1.2-3.el9.x86_64 100/121 2024-04-26T04:23:33.842 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : tracker-miners-3.1.2-4.el9.x86_64 101/121 2024-04-26T04:23:33.860 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: tracker-miners-3.1.2-4.el9.x86_64 101/121 2024-04-26T04:23:34.084 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : vulkan-loader-1.3.268.0-1.el9.x86_64 102/121 2024-04-26T04:23:34.113 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : mesa-vulkan-drivers-23.3.3-1.el9.x86_64 103/121 2024-04-26T04:23:34.139 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : bubblewrap-0.4.1-7.el9.x86_64 104/121 2024-04-26T04:23:34.162 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : bluez-libs-5.56-6.el9.x86_64 105/121 2024-04-26T04:23:34.608 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: pipewire-1.0.1-1.el9.x86_64 106/121 2024-04-26T04:23:34.633 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-1.0.1-1.el9.x86_64 106/121 2024-04-26T04:23:34.648 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: pipewire-1.0.1-1.el9.x86_64 106/121 2024-04-26T04:23:34.648 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire.socket → /usr/lib/systemd/user/pipewire.socket. 2024-04-26T04:23:34.648 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:34.885 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-libs-1.0.1-1.el9.x86_64 107/121 2024-04-26T04:23:34.947 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : wireplumber-0.4.14-1.el9.x86_64 108/121 2024-04-26T04:23:34.987 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : wireplumber-libs-0.4.14-1.el9.x86_64 109/121 2024-04-26T04:23:35.031 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-jack-audio-connection-kit-libs-1.0.1-1. 110/121 2024-04-26T04:23:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:34 smithi098 ceph-mon[27184]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:35.056 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : ModemManager-glib-1.20.2-1.el9.x86_64 111/121 2024-04-26T04:23:35.497 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 112/121 2024-04-26T04:23:35.545 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : geoclue2-2.6.0-7.el9.x86_64 112/121 2024-04-26T04:23:35.561 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 112/121 2024-04-26T04:23:36.090 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: flatpak-1.12.8-1.el9.x86_64 113/121 2024-04-26T04:23:36.183 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : flatpak-1.12.8-1.el9.x86_64 113/121 2024-04-26T04:23:36.228 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xdg-desktop-portal-1.12.6-1.el9.x86_64 114/121 2024-04-26T04:23:36.255 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: xdg-desktop-portal-1.12.6-1.el9.x86_64 114/121 2024-04-26T04:23:36.432 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libcanberra-gtk3-0.30-27.el9.x86_64 115/121 2024-04-26T04:23:36.486 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gtk3-3.24.31-2.el9.x86_64 116/121 2024-04-26T04:23:36.512 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 117/121 2024-04-26T04:23:36.524 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 117/121 2024-04-26T04:23:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:36 smithi098 ceph-mon[27184]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:38 smithi098 ceph-mon[27184]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:40 smithi098 ceph-mon[27184]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:42 smithi098 ceph-mon[27184]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:44.290 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: google-chrome-stable-124.0.6367.91-1.x86_64 118/121 2024-04-26T04:23:44.322 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : google-chrome-stable-124.0.6367.91-1.x86_64 118/121 2024-04-26T04:23:44.539 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: google-chrome-stable-124.0.6367.91-1.x86_64 118/121 2024-04-26T04:23:44.578 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-jack-audio-connection-kit-1.0.1-1.el9.x 119/121 2024-04-26T04:23:44.626 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-alsa-1.0.1-1.el9.x86_64 120/121 2024-04-26T04:23:44.650 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pipewire-pulseaudio-1.0.1-1.el9.x86_64 121/121 2024-04-26T04:23:44.666 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: pipewire-pulseaudio-1.0.1-1.el9.x86_64 121/121 2024-04-26T04:23:44.666 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire-pulse.socket → /usr/lib/systemd/user/pipewire-pulse.socket. 2024-04-26T04:23:44.666 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:44.687 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 121/121 2024-04-26T04:23:44.698 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: wireplumber-0.4.14-1.el9.x86_64 121/121 2024-04-26T04:23:44.698 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/user/pipewire-session-manager.service → /usr/lib/systemd/user/wireplumber.service. 2024-04-26T04:23:44.698 INFO:tasks.workunit.client.0.smithi098.stdout:Created symlink /etc/systemd/user/pipewire.service.wants/wireplumber.service → /usr/lib/systemd/user/wireplumber.service. 2024-04-26T04:23:44.698 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:44 smithi098 ceph-mon[27184]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:46 smithi098 ceph-mon[27184]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:48 smithi098 ceph-mon[27184]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: pipewire-pulseaudio-1.0.1-1.el9.x86_64 121/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : ModemManager-glib-1.20.2-1.el9.x86_64 1/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : bluez-libs-5.56-6.el9.x86_64 2/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : bubblewrap-0.4.1-7.el9.x86_64 3/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : cups-libs-1:2.3.3op2-24.el9.x86_64 4/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libpciaccess-0.16-7.el9.x86_64 5/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : adwaita-cursor-theme-40.1.1-3.el9.noarch 6/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : adwaita-icon-theme-40.1.1-3.el9.noarch 7/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : alsa-lib-1.2.10-2.el9.x86_64 8/121 2024-04-26T04:23:50.829 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : at-spi2-atk-2.38.0-4.el9.x86_64 9/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : at-spi2-core-2.40.3-1.el9.x86_64 10/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : atk-2.36.0-5.el9.x86_64 11/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : avahi-glib-0.8-20.el9.x86_64 12/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : colord-libs-1.4.5-4.el9.x86_64 13/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : dconf-0.40.0-6.el9.x86_64 14/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 15/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : exiv2-0.27.5-2.el9.x86_64 16/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : exiv2-libs-0.27.5-2.el9.x86_64 17/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : fdk-aac-free-2.0.0-8.el9.x86_64 18/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : flac-libs-1.3.3-12.el9.x86_64 19/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : flatpak-1.12.8-1.el9.x86_64 20/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : flatpak-selinux-1.12.8-1.el9.noarch 21/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : flatpak-session-helper-1.12.8-1.el9.x86_64 22/121 2024-04-26T04:23:50.830 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 23/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : geoclue2-2.6.0-7.el9.x86_64 24/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : giflib-5.2.1-9.el9.x86_64 25/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : graphene-1.10.6-2.el9.x86_64 26/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gsm-1.0.19-6.el9.x86_64 27/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gstreamer1-1.22.1-2.el9.x86_64 28/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gstreamer1-plugins-base-1.22.1-2.el9.x86_64 29/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gtk-update-icon-cache-3.24.31-2.el9.x86_64 30/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gtk3-3.24.31-2.el9.x86_64 31/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : hicolor-icon-theme-0.17-13.el9.noarch 32/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : iso-codes-4.6.0-3.el9.noarch 33/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : jbigkit-libs-2.1-23.el9.x86_64 34/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : lcms2-2.12-3.el9.x86_64 35/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libX11-xcb-1.7.0-9.el9.x86_64 36/121 2024-04-26T04:23:50.831 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXcomposite-0.4.5-7.el9.x86_64 37/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXcursor-1.2.0-7.el9.x86_64 38/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXdamage-1.1.5-7.el9.x86_64 39/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXfixes-5.0.3-16.el9.x86_64 40/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXi-1.7.10-8.el9.x86_64 41/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXinerama-1.1.4-10.el9.x86_64 42/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXrandr-1.5.2-8.el9.x86_64 43/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXtst-1.2.3-16.el9.x86_64 44/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXv-1.0.11-16.el9.x86_64 45/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXxf86vm-1.1.4-18.el9.x86_64 46/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libasyncns-0.8-22.el9.x86_64 47/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libcanberra-0.30-27.el9.x86_64 48/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libcanberra-gtk3-0.30-27.el9.x86_64 49/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libdrm-2.4.117-1.el9.x86_64 50/121 2024-04-26T04:23:50.832 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libepoxy-1.5.5-4.el9.x86_64 51/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : liberation-fonts-1:2.1.3-5.el9.noarch 52/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : liberation-fonts-common-1:2.1.3-5.el9.noarch 53/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : liberation-mono-fonts-1:2.1.3-5.el9.noarch 54/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : liberation-sans-fonts-1:2.1.3-5.el9.noarch 55/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : liberation-serif-fonts-1:2.1.3-5.el9.noarch 56/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libexif-0.6.22-6.el9.x86_64 57/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libgexiv2-0.12.3-1.el9.x86_64 58/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-1:1.3.4-1.el9.x86_64 59/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-egl-1:1.3.4-1.el9.x86_64 60/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-glx-1:1.3.4-1.el9.x86_64 61/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libgsf-1.14.47-5.el9.x86_64 62/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libgxps-0.3.2-3.el9.x86_64 63/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libiptcdata-1.0.5-9.el9.x86_64 64/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libldac-2.0.2.3-10.el9.x86_64 65/121 2024-04-26T04:23:50.833 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libnotify-0.7.9-8.el9.x86_64 66/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libogg-2:1.3.4-6.el9.x86_64 67/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libosinfo-1.10.0-1.el9.x86_64 68/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libsbc-1.4-9.el9.x86_64 69/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libsndfile-1.0.31-8.el9.x86_64 70/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libtheora-1:1.1.1-31.el9.x86_64 71/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libtiff-4.4.0-12.el9.x86_64 72/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libtool-ltdl-2.4.6-45.el9.x86_64 73/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libtracker-sparql-3.1.2-3.el9.x86_64 74/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libvisual-1:0.4.0-34.el9.x86_64 75/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libvorbis-1:1.3.7-5.el9.x86_64 76/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libwayland-client-1.21.0-1.el9.x86_64 77/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libwayland-cursor-1.21.0-1.el9.x86_64 78/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libwayland-egl-1.21.0-1.el9.x86_64 79/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libwayland-server-1.21.0-1.el9.x86_64 80/121 2024-04-26T04:23:50.834 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libwebp-1.2.0-8.el9.x86_64 81/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxkbcommon-1.0.3-4.el9.x86_64 82/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxshmfence-1.3-10.el9.x86_64 83/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : low-memory-monitor-2.1-4.el9.x86_64 84/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : mesa-libEGL-23.3.3-1.el9.x86_64 85/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : mesa-libGL-23.3.3-1.el9.x86_64 86/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : mesa-libgbm-23.3.3-1.el9.x86_64 87/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : mesa-libglapi-23.3.3-1.el9.x86_64 88/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : mesa-vulkan-drivers-23.3.3-1.el9.x86_64 89/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : openjpeg2-2.4.0-7.el9.x86_64 90/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : opus-1.3.1-10.el9.x86_64 91/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : orc-0.4.31-6.el9.x86_64 92/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : osinfo-db-20231215-1.el9.noarch 93/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : osinfo-db-tools-1.10.0-1.el9.x86_64 94/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : ostree-libs-2024.5-2.el9.x86_64 95/121 2024-04-26T04:23:50.835 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : p11-kit-server-0.25.3-2.el9.x86_64 96/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-1.0.1-1.el9.x86_64 97/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-alsa-1.0.1-1.el9.x86_64 98/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-jack-audio-connection-kit-1.0.1-1.el9.x 99/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-jack-audio-connection-kit-libs-1.0.1-1. 100/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-libs-1.0.1-1.el9.x86_64 101/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pipewire-pulseaudio-1.0.1-1.el9.x86_64 102/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : poppler-21.01.0-19.el9.x86_64 103/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : poppler-data-0.4.9-9.el9.noarch 104/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : poppler-glib-21.01.0-19.el9.x86_64 105/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pulseaudio-libs-15.0-2.el9.x86_64 106/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : rtkit-0.11-28.el9.x86_64 107/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : sound-theme-freedesktop-0.8-17.el9.noarch 108/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : totem-pl-parser-3.26.6-2.el9.x86_64 109/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : tracker-3.1.2-3.el9.x86_64 110/121 2024-04-26T04:23:50.836 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : tracker-miners-3.1.2-4.el9.x86_64 111/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : upower-0.99.11-11.el9.x86_64 112/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : vulkan-loader-1.3.268.0-1.el9.x86_64 113/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : webrtc-audio-processing-0.3.1-8.el9.x86_64 114/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : wireplumber-0.4.14-1.el9.x86_64 115/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : wireplumber-libs-0.4.14-1.el9.x86_64 116/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xdg-dbus-proxy-0.1.3-1.el9.x86_64 117/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xdg-desktop-portal-1.12.6-1.el9.x86_64 118/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 119/121 2024-04-26T04:23:50.837 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xkeyboard-config-2.33-2.el9.noarch 120/121 2024-04-26T04:23:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:50 smithi098 ceph-mon[27184]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:52.736 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : google-chrome-stable-124.0.6367.91-1.x86_64 121/121 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout:Installed: 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: ModemManager-glib-1.20.2-1.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: adwaita-cursor-theme-40.1.1-3.el9.noarch 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: adwaita-icon-theme-40.1.1-3.el9.noarch 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: alsa-lib-1.2.10-2.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-atk-2.38.0-4.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-core-2.40.3-1.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: atk-2.36.0-5.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: avahi-glib-0.8-20.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: bluez-libs-5.56-6.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: bubblewrap-0.4.1-7.el9.x86_64 2024-04-26T04:23:52.737 INFO:tasks.workunit.client.0.smithi098.stdout: colord-libs-1.4.5-4.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: cups-libs-1:2.3.3op2-24.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: dconf-0.40.0-6.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: exiv2-0.27.5-2.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: exiv2-libs-0.27.5-2.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: fdk-aac-free-2.0.0-8.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: flac-libs-1.3.3-12.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak-1.12.8-1.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak-selinux-1.12.8-1.el9.noarch 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: flatpak-session-helper-1.12.8-1.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: geoclue2-2.6.0-7.el9.x86_64 2024-04-26T04:23:52.738 INFO:tasks.workunit.client.0.smithi098.stdout: giflib-5.2.1-9.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: google-chrome-stable-124.0.6367.91-1.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: graphene-1.10.6-2.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: gsm-1.0.19-6.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: gstreamer1-1.22.1-2.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: gstreamer1-plugins-base-1.22.1-2.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: gtk-update-icon-cache-3.24.31-2.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: gtk3-3.24.31-2.el9.x86_64 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: hicolor-icon-theme-0.17-13.el9.noarch 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: iso-codes-4.6.0-3.el9.noarch 2024-04-26T04:23:52.739 INFO:tasks.workunit.client.0.smithi098.stdout: jbigkit-libs-2.1-23.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: lcms2-2.12-3.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libX11-xcb-1.7.0-9.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libXcomposite-0.4.5-7.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libXcursor-1.2.0-7.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libXdamage-1.1.5-7.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libXfixes-5.0.3-16.el9.x86_64 2024-04-26T04:23:52.740 INFO:tasks.workunit.client.0.smithi098.stdout: libXi-1.7.10-8.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libXinerama-1.1.4-10.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libXrandr-1.5.2-8.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libXtst-1.2.3-16.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libXv-1.0.11-16.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libXxf86vm-1.1.4-18.el9.x86_64 2024-04-26T04:23:52.741 INFO:tasks.workunit.client.0.smithi098.stdout: libasyncns-0.8-22.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra-0.30-27.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra-gtk3-0.30-27.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libdrm-2.4.117-1.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libepoxy-1.5.5-4.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-fonts-1:2.1.3-5.el9.noarch 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-fonts-common-1:2.1.3-5.el9.noarch 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-mono-fonts-1:2.1.3-5.el9.noarch 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-sans-fonts-1:2.1.3-5.el9.noarch 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: liberation-serif-fonts-1:2.1.3-5.el9.noarch 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libexif-0.6.22-6.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libgexiv2-0.12.3-1.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-1:1.3.4-1.el9.x86_64 2024-04-26T04:23:52.742 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-egl-1:1.3.4-1.el9.x86_64 2024-04-26T04:23:52.743 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-glx-1:1.3.4-1.el9.x86_64 2024-04-26T04:23:52.743 INFO:tasks.workunit.client.0.smithi098.stdout: libgsf-1.14.47-5.el9.x86_64 2024-04-26T04:23:52.743 INFO:tasks.workunit.client.0.smithi098.stdout: libgxps-0.3.2-3.el9.x86_64 2024-04-26T04:23:52.743 INFO:tasks.workunit.client.0.smithi098.stdout: libiptcdata-1.0.5-9.el9.x86_64 2024-04-26T04:23:52.744 INFO:tasks.workunit.client.0.smithi098.stdout: libldac-2.0.2.3-10.el9.x86_64 2024-04-26T04:23:52.744 INFO:tasks.workunit.client.0.smithi098.stdout: libnotify-0.7.9-8.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libogg-2:1.3.4-6.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libosinfo-1.10.0-1.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libpciaccess-0.16-7.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libsbc-1.4-9.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libsndfile-1.0.31-8.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libtheora-1:1.1.1-31.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libtiff-4.4.0-12.el9.x86_64 2024-04-26T04:23:52.745 INFO:tasks.workunit.client.0.smithi098.stdout: libtool-ltdl-2.4.6-45.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libtracker-sparql-3.1.2-3.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libvisual-1:0.4.0-34.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libvorbis-1:1.3.7-5.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-client-1.21.0-1.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-cursor-1.21.0-1.el9.x86_64 2024-04-26T04:23:52.746 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-egl-1.21.0-1.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: libwayland-server-1.21.0-1.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: libwebp-1.2.0-8.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbcommon-1.0.3-4.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: libxshmfence-1.3-10.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: low-memory-monitor-2.1-4.el9.x86_64 2024-04-26T04:23:52.747 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libEGL-23.3.3-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libGL-23.3.3-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libgbm-23.3.3-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-libglapi-23.3.3-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: mesa-vulkan-drivers-23.3.3-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: openjpeg2-2.4.0-7.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: opus-1.3.1-10.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: orc-0.4.31-6.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: osinfo-db-20231215-1.el9.noarch 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: osinfo-db-tools-1.10.0-1.el9.x86_64 2024-04-26T04:23:52.748 INFO:tasks.workunit.client.0.smithi098.stdout: ostree-libs-2024.5-2.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: p11-kit-server-0.25.3-2.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-alsa-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-jack-audio-connection-kit-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-jack-audio-connection-kit-libs-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-libs-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pipewire-pulseaudio-1.0.1-1.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: poppler-21.01.0-19.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: poppler-data-0.4.9-9.el9.noarch 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: poppler-glib-21.01.0-19.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: pulseaudio-libs-15.0-2.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: rtkit-0.11-28.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: sound-theme-freedesktop-0.8-17.el9.noarch 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: totem-pl-parser-3.26.6-2.el9.x86_64 2024-04-26T04:23:52.749 INFO:tasks.workunit.client.0.smithi098.stdout: tracker-3.1.2-3.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: tracker-miners-3.1.2-4.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: upower-0.99.11-11.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: vulkan-loader-1.3.268.0-1.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: webrtc-audio-processing-0.3.1-8.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: wireplumber-0.4.14-1.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: wireplumber-libs-0.4.14-1.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-dbus-proxy-0.1.3-1.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-desktop-portal-1.12.6-1.el9.x86_64 2024-04-26T04:23:52.750 INFO:tasks.workunit.client.0.smithi098.stdout: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 2024-04-26T04:23:52.751 INFO:tasks.workunit.client.0.smithi098.stdout: xkeyboard-config-2.33-2.el9.noarch 2024-04-26T04:23:52.751 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:52.751 INFO:tasks.workunit.client.0.smithi098.stdout:Complete! 2024-04-26T04:23:53.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:52 smithi098 ceph-mon[27184]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:53.159 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo rm /etc/yum.repos.d/google-chrome.repo 2024-04-26T04:23:53.172 INFO:tasks.workunit.client.0.smithi098.stderr:+ sudo yum install -y xorg-x11-server-Xvfb gtk2-devel gtk3-devel libnotify-devel GConf2 nss.x86_64 libXScrnSaver alsa-lib 2024-04-26T04:23:53.711 INFO:tasks.workunit.client.0.smithi098.stdout:Last metadata expiration check: 0:09:36 ago on Fri 26 Apr 2024 04:14:17 AM UTC. 2024-04-26T04:23:53.801 INFO:tasks.workunit.client.0.smithi098.stdout:Package nss-3.90.0-6.el9.x86_64 is already installed. 2024-04-26T04:23:53.803 INFO:tasks.workunit.client.0.smithi098.stdout:Package alsa-lib-1.2.10-2.el9.x86_64 is already installed. 2024-04-26T04:23:53.860 INFO:tasks.workunit.client.0.smithi098.stdout:Dependencies resolved. 2024-04-26T04:23:53.866 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:23:53.866 INFO:tasks.workunit.client.0.smithi098.stdout: Package Arch Version Repository Size 2024-04-26T04:23:53.866 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:23:53.866 INFO:tasks.workunit.client.0.smithi098.stdout:Installing: 2024-04-26T04:23:53.867 INFO:tasks.workunit.client.0.smithi098.stdout: GConf2 x86_64 3.2.6-34.el9 epel 979 k 2024-04-26T04:23:53.867 INFO:tasks.workunit.client.0.smithi098.stdout: gtk2-devel x86_64 2.24.33-8.el9 appstream 2.8 M 2024-04-26T04:23:53.867 INFO:tasks.workunit.client.0.smithi098.stdout: gtk3-devel x86_64 3.24.31-2.el9 appstream 4.3 M 2024-04-26T04:23:53.867 INFO:tasks.workunit.client.0.smithi098.stdout: libXScrnSaver x86_64 1.2.3-10.el9 appstream 25 k 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout: libnotify-devel x86_64 0.7.9-8.el9 appstream 42 k 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-server-Xvfb x86_64 1.20.11-24.el9 appstream 894 k 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout:Upgrading: 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout: nss x86_64 3.90.0-7.el9 appstream 706 k 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout: nss-sysinit x86_64 3.90.0-7.el9 appstream 20 k 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout:Installing dependencies: 2024-04-26T04:23:53.870 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-atk-devel x86_64 2.38.0-4.el9 appstream 11 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-core-devel x86_64 2.40.3-1.el9 appstream 148 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: atk-devel x86_64 2.36.0-5.el9 appstream 190 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: brotli x86_64 1.0.9-6.el9 appstream 313 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: brotli-devel x86_64 1.0.9-6.el9 appstream 32 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: bzip2-devel x86_64 1.0.8-8.el9 appstream 215 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: cairo-devel x86_64 1.17.4-7.el9 appstream 209 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: cairo-gobject-devel x86_64 1.17.4-7.el9 appstream 11 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: cmake-filesystem x86_64 3.26.5-2.el9 appstream 19 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: dbus-devel x86_64 1:1.12.20-8.el9 appstream 37 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: dbus-glib x86_64 0.110-13.el9 appstream 123 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: fontconfig-devel x86_64 2.14.0-2.el9 appstream 174 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: freetype-devel x86_64 2.10.4-9.el9 appstream 1.1 M 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: fribidi-devel x86_64 1.0.10-6.el9.2 appstream 27 k 2024-04-26T04:23:53.871 INFO:tasks.workunit.client.0.smithi098.stdout: gdk-pixbuf2-devel x86_64 2.42.6-3.el9 appstream 67 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: glib2-devel x86_64 2.68.4-14.el9 appstream 552 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: graphite2-devel x86_64 1.3.14-9.el9 appstream 22 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: gtk2 x86_64 2.24.33-8.el9 appstream 3.5 M 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: harfbuzz-devel x86_64 2.7.4-10.el9 appstream 347 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: harfbuzz-icu x86_64 2.7.4-10.el9 appstream 14 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libICE x86_64 1.0.10-8.el9 appstream 71 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libSM x86_64 1.2.3-10.el9 appstream 42 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libX11-devel x86_64 1.7.0-9.el9 appstream 1.1 M 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXau-devel x86_64 1.0.9-8.el9 appstream 14 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXcomposite-devel x86_64 0.4.5-7.el9 appstream 17 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXcursor-devel x86_64 1.2.0-7.el9 appstream 26 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXdamage-devel x86_64 1.1.5-7.el9 appstream 10 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXdmcp x86_64 1.1.3-8.el9 appstream 36 k 2024-04-26T04:23:53.872 INFO:tasks.workunit.client.0.smithi098.stdout: libXext-devel x86_64 1.3.4-8.el9 appstream 87 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXfixes-devel x86_64 5.0.3-16.el9 appstream 13 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXfont2 x86_64 2.0.3-12.el9 appstream 143 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXft-devel x86_64 2.3.3-8.el9 appstream 19 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXi-devel x86_64 1.7.10-8.el9 appstream 113 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXinerama-devel x86_64 1.1.4-10.el9 appstream 14 k 2024-04-26T04:23:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: libXmu x86_64 1.1.3-8.el9 appstream 75 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libXrandr-devel x86_64 1.5.2-8.el9 appstream 20 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libXrender-devel x86_64 0.9.10-16.el9 appstream 17 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libXt x86_64 1.2.0-6.el9 appstream 180 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libXtst-devel x86_64 1.2.3-16.el9 appstream 17 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libblkid-devel x86_64 2.37.4-18.el9 appstream 18 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libdatrie-devel x86_64 0.2.13-4.el9 appstream 142 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libepoxy-devel x86_64 1.5.5-4.el9 appstream 134 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libffi-devel x86_64 3.4.2-8.el9 appstream 29 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libfontenc x86_64 1.1.3-17.el9 appstream 31 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-core-devel x86_64 1:1.3.4-1.el9 appstream 18 k 2024-04-26T04:23:53.874 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-devel x86_64 1:1.3.4-1.el9 appstream 161 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-gles x86_64 1:1.3.4-1.el9 appstream 33 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-opengl x86_64 1:1.3.4-1.el9 appstream 43 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libicu-devel x86_64 67.1-9.el9 appstream 938 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libmount-devel x86_64 2.37.4-18.el9 appstream 19 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libpng-devel x86_64 2:1.6.37-12.el9 appstream 293 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libselinux-devel x86_64 3.6-1.el9 appstream 159 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libsepol-devel x86_64 3.6-1.el9 appstream 47 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libthai-devel x86_64 0.1.28-8.el9 appstream 134 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libtiff-devel x86_64 4.4.0-12.el9 appstream 556 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libxcb-devel x86_64 1.13.1-9.el9 appstream 1.5 M 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbcommon-devel x86_64 1.0.3-4.el9 appstream 62 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbfile x86_64 1.1.0-8.el9 appstream 89 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: libxml2-devel x86_64 2.9.13-5.el9 appstream 898 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: pango-devel x86_64 1.48.7-3.el9 appstream 149 k 2024-04-26T04:23:53.875 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-cpp x86_64 8.44-3.el9.3 appstream 27 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-devel x86_64 8.44-3.el9.3 appstream 507 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-utf16 x86_64 8.44-3.el9.3 appstream 185 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-utf32 x86_64 8.44-3.el9.3 appstream 176 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-devel x86_64 10.40-5.el9 appstream 512 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-utf16 x86_64 10.40-5.el9 appstream 214 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-utf32 x86_64 10.40-5.el9 appstream 203 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: pixman-devel x86_64 0.40.0-6.el9 appstream 17 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: sysprof-capture-devel x86_64 3.40.1-3.el9 appstream 60 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: wayland-devel x86_64 1.21.0-1.el9 appstream 145 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: xkbcomp x86_64 1.4.4-4.el9 appstream 100 k 2024-04-26T04:23:53.876 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-proto-devel noarch 2022.2-1.el9 appstream 309 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-server-common x86_64 1.20.11-24.el9 appstream 36 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-xauth x86_64 1:1.1-10.el9 appstream 37 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: xz-devel x86_64 5.2.5-8.el9 appstream 55 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout:Installing weak dependencies: 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra-gtk2 x86_64 0.30-27.el9 appstream 26 k 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction Summary 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout:Install 83 Packages 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout:Upgrade 2 Packages 2024-04-26T04:23:53.877 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:23:53.878 INFO:tasks.workunit.client.0.smithi098.stdout:Total download size: 27 M 2024-04-26T04:23:53.878 INFO:tasks.workunit.client.0.smithi098.stdout:Downloading Packages: 2024-04-26T04:23:54.590 INFO:tasks.workunit.client.0.smithi098.stdout:(1/85): at-spi2-atk-devel-2.38.0-4.el9.x86_64.r 51 kB/s | 11 kB 00:00 2024-04-26T04:23:54.624 INFO:tasks.workunit.client.0.smithi098.stdout:(2/85): at-spi2-core-devel-2.40.3-1.el9.x86_64. 613 kB/s | 148 kB 00:00 2024-04-26T04:23:54.649 INFO:tasks.workunit.client.0.smithi098.stdout:(3/85): atk-devel-2.36.0-5.el9.x86_64.rpm 712 kB/s | 190 kB 00:00 2024-04-26T04:23:54.758 INFO:tasks.workunit.client.0.smithi098.stdout:(4/85): brotli-1.0.9-6.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-04-26T04:23:54.783 INFO:tasks.workunit.client.0.smithi098.stdout:(5/85): brotli-devel-1.0.9-6.el9.x86_64.rpm 204 kB/s | 32 kB 00:00 2024-04-26T04:23:54.808 INFO:tasks.workunit.client.0.smithi098.stdout:(6/85): bzip2-devel-1.0.8-8.el9.x86_64.rpm 1.3 MB/s | 215 kB 00:00 2024-04-26T04:23:54.834 INFO:tasks.workunit.client.0.smithi098.stdout:(7/85): cairo-devel-1.17.4-7.el9.x86_64.rpm 2.7 MB/s | 209 kB 00:00 2024-04-26T04:23:54.859 INFO:tasks.workunit.client.0.smithi098.stdout:(8/85): cairo-gobject-devel-1.17.4-7.el9.x86_64 227 kB/s | 11 kB 00:00 2024-04-26T04:23:54.884 INFO:tasks.workunit.client.0.smithi098.stdout:(9/85): cmake-filesystem-3.26.5-2.el9.x86_64.rp 251 kB/s | 19 kB 00:00 2024-04-26T04:23:54.910 INFO:tasks.workunit.client.0.smithi098.stdout:(10/85): dbus-devel-1.12.20-8.el9.x86_64.rpm 488 kB/s | 37 kB 00:00 2024-04-26T04:23:54.935 INFO:tasks.workunit.client.0.smithi098.stdout:(11/85): dbus-glib-0.110-13.el9.x86_64.rpm 1.6 MB/s | 123 kB 00:00 2024-04-26T04:23:54.960 INFO:tasks.workunit.client.0.smithi098.stdout:(12/85): fontconfig-devel-2.14.0-2.el9.x86_64.r 3.4 MB/s | 174 kB 00:00 2024-04-26T04:23:55.011 INFO:tasks.workunit.client.0.smithi098.stdout:(13/85): freetype-devel-2.10.4-9.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-04-26T04:23:55.036 INFO:tasks.workunit.client.0.smithi098.stdout:(14/85): fribidi-devel-1.0.10-6.el9.2.x86_64.rp 273 kB/s | 27 kB 00:00 2024-04-26T04:23:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:54 smithi098 ceph-mon[27184]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:55.061 INFO:tasks.workunit.client.0.smithi098.stdout:(15/85): gdk-pixbuf2-devel-2.42.6-3.el9.x86_64. 670 kB/s | 67 kB 00:00 2024-04-26T04:23:55.086 INFO:tasks.workunit.client.0.smithi098.stdout:(16/85): glib2-devel-2.68.4-14.el9.x86_64.rpm 7.2 MB/s | 552 kB 00:00 2024-04-26T04:23:55.170 INFO:tasks.workunit.client.0.smithi098.stdout:(17/85): graphite2-devel-1.3.14-9.el9.x86_64.rp 202 kB/s | 22 kB 00:00 2024-04-26T04:23:55.229 INFO:tasks.workunit.client.0.smithi098.stdout:(18/85): gtk2-2.24.33-8.el9.x86_64.rpm 21 MB/s | 3.5 MB 00:00 2024-04-26T04:23:55.304 INFO:tasks.workunit.client.0.smithi098.stdout:(19/85): gtk2-devel-2.24.33-8.el9.x86_64.rpm 13 MB/s | 2.8 MB 00:00 2024-04-26T04:23:55.446 INFO:tasks.workunit.client.0.smithi098.stdout:(20/85): gtk3-devel-3.24.31-2.el9.x86_64.rpm 16 MB/s | 4.3 MB 00:00 2024-04-26T04:23:55.472 INFO:tasks.workunit.client.0.smithi098.stdout:(21/85): harfbuzz-devel-2.7.4-10.el9.x86_64.rpm 1.4 MB/s | 347 kB 00:00 2024-04-26T04:23:55.497 INFO:tasks.workunit.client.0.smithi098.stdout:(22/85): harfbuzz-icu-2.7.4-10.el9.x86_64.rpm 72 kB/s | 14 kB 00:00 2024-04-26T04:23:55.522 INFO:tasks.workunit.client.0.smithi098.stdout:(23/85): libICE-1.0.10-8.el9.x86_64.rpm 951 kB/s | 71 kB 00:00 2024-04-26T04:23:55.556 INFO:tasks.workunit.client.0.smithi098.stdout:(24/85): libSM-1.2.3-10.el9.x86_64.rpm 720 kB/s | 42 kB 00:00 2024-04-26T04:23:55.598 INFO:tasks.workunit.client.0.smithi098.stdout:(25/85): libX11-devel-1.7.0-9.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-04-26T04:23:55.623 INFO:tasks.workunit.client.0.smithi098.stdout:(26/85): libXScrnSaver-1.2.3-10.el9.x86_64.rpm 249 kB/s | 25 kB 00:00 2024-04-26T04:23:55.648 INFO:tasks.workunit.client.0.smithi098.stdout:(27/85): libXau-devel-1.0.9-8.el9.x86_64.rpm 158 kB/s | 14 kB 00:00 2024-04-26T04:23:55.673 INFO:tasks.workunit.client.0.smithi098.stdout:(28/85): libXcomposite-devel-0.4.5-7.el9.x86_64 219 kB/s | 17 kB 00:00 2024-04-26T04:23:55.707 INFO:tasks.workunit.client.0.smithi098.stdout:(29/85): libXcursor-devel-1.2.0-7.el9.x86_64.rp 451 kB/s | 26 kB 00:00 2024-04-26T04:23:55.733 INFO:tasks.workunit.client.0.smithi098.stdout:(30/85): libXdamage-devel-1.1.5-7.el9.x86_64.rp 124 kB/s | 10 kB 00:00 2024-04-26T04:23:55.758 INFO:tasks.workunit.client.0.smithi098.stdout:(31/85): libXdmcp-1.1.3-8.el9.x86_64.rpm 423 kB/s | 36 kB 00:00 2024-04-26T04:23:55.783 INFO:tasks.workunit.client.0.smithi098.stdout:(32/85): libXext-devel-1.3.4-8.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-04-26T04:23:55.809 INFO:tasks.workunit.client.0.smithi098.stdout:(33/85): libXfixes-devel-5.0.3-16.el9.x86_64.rp 176 kB/s | 13 kB 00:00 2024-04-26T04:23:55.834 INFO:tasks.workunit.client.0.smithi098.stdout:(34/85): libXfont2-2.0.3-12.el9.x86_64.rpm 1.9 MB/s | 143 kB 00:00 2024-04-26T04:23:55.859 INFO:tasks.workunit.client.0.smithi098.stdout:(35/85): libXft-devel-2.3.3-8.el9.x86_64.rpm 254 kB/s | 19 kB 00:00 2024-04-26T04:23:55.885 INFO:tasks.workunit.client.0.smithi098.stdout:(36/85): libXi-devel-1.7.10-8.el9.x86_64.rpm 1.5 MB/s | 113 kB 00:00 2024-04-26T04:23:55.910 INFO:tasks.workunit.client.0.smithi098.stdout:(37/85): libXinerama-devel-1.1.4-10.el9.x86_64. 184 kB/s | 14 kB 00:00 2024-04-26T04:23:55.935 INFO:tasks.workunit.client.0.smithi098.stdout:(38/85): libXmu-1.1.3-8.el9.x86_64.rpm 991 kB/s | 75 kB 00:00 2024-04-26T04:23:55.960 INFO:tasks.workunit.client.0.smithi098.stdout:(39/85): libXrandr-devel-1.5.2-8.el9.x86_64.rpm 265 kB/s | 20 kB 00:00 2024-04-26T04:23:55.986 INFO:tasks.workunit.client.0.smithi098.stdout:(40/85): libXrender-devel-0.9.10-16.el9.x86_64. 220 kB/s | 17 kB 00:00 2024-04-26T04:23:56.011 INFO:tasks.workunit.client.0.smithi098.stdout:(41/85): libXt-1.2.0-6.el9.x86_64.rpm 2.3 MB/s | 180 kB 00:00 2024-04-26T04:23:56.036 INFO:tasks.workunit.client.0.smithi098.stdout:(42/85): libXtst-devel-1.2.3-16.el9.x86_64.rpm 219 kB/s | 17 kB 00:00 2024-04-26T04:23:56.062 INFO:tasks.workunit.client.0.smithi098.stdout:(43/85): libblkid-devel-2.37.4-18.el9.x86_64.rp 235 kB/s | 18 kB 00:00 2024-04-26T04:23:56.087 INFO:tasks.workunit.client.0.smithi098.stdout:(44/85): libcanberra-gtk2-0.30-27.el9.x86_64.rp 343 kB/s | 26 kB 00:00 2024-04-26T04:23:56.121 INFO:tasks.workunit.client.0.smithi098.stdout:(45/85): libdatrie-devel-0.2.13-4.el9.x86_64.rp 1.6 MB/s | 142 kB 00:00 2024-04-26T04:23:56.146 INFO:tasks.workunit.client.0.smithi098.stdout:(46/85): libepoxy-devel-1.5.5-4.el9.x86_64.rpm 1.6 MB/s | 134 kB 00:00 2024-04-26T04:23:56.172 INFO:tasks.workunit.client.0.smithi098.stdout:(47/85): libffi-devel-3.4.2-8.el9.x86_64.rpm 349 kB/s | 29 kB 00:00 2024-04-26T04:23:56.197 INFO:tasks.workunit.client.0.smithi098.stdout:(48/85): libfontenc-1.1.3-17.el9.x86_64.rpm 412 kB/s | 31 kB 00:00 2024-04-26T04:23:56.222 INFO:tasks.workunit.client.0.smithi098.stdout:(49/85): libglvnd-core-devel-1.3.4-1.el9.x86_64 241 kB/s | 18 kB 00:00 2024-04-26T04:23:56.247 INFO:tasks.workunit.client.0.smithi098.stdout:(50/85): libglvnd-devel-1.3.4-1.el9.x86_64.rpm 2.1 MB/s | 161 kB 00:00 2024-04-26T04:23:56.273 INFO:tasks.workunit.client.0.smithi098.stdout:(51/85): libglvnd-gles-1.3.4-1.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-04-26T04:23:56.298 INFO:tasks.workunit.client.0.smithi098.stdout:(52/85): libglvnd-opengl-1.3.4-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-04-26T04:23:56.357 INFO:tasks.workunit.client.0.smithi098.stdout:(53/85): libicu-devel-67.1-9.el9.x86_64.rpm 8.4 MB/s | 938 kB 00:00 2024-04-26T04:23:56.382 INFO:tasks.workunit.client.0.smithi098.stdout:(54/85): libmount-devel-2.37.4-18.el9.x86_64.rp 175 kB/s | 19 kB 00:00 2024-04-26T04:23:56.407 INFO:tasks.workunit.client.0.smithi098.stdout:(55/85): libnotify-devel-0.7.9-8.el9.x86_64.rpm 390 kB/s | 42 kB 00:00 2024-04-26T04:23:56.433 INFO:tasks.workunit.client.0.smithi098.stdout:(56/85): libpng-devel-1.6.37-12.el9.x86_64.rpm 3.8 MB/s | 293 kB 00:00 2024-04-26T04:23:56.475 INFO:tasks.workunit.client.0.smithi098.stdout:(57/85): libsepol-devel-3.6-1.el9.x86_64.rpm 714 kB/s | 47 kB 00:00 2024-04-26T04:23:56.500 INFO:tasks.workunit.client.0.smithi098.stdout:(58/85): libselinux-devel-3.6-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-04-26T04:23:56.525 INFO:tasks.workunit.client.0.smithi098.stdout:(59/85): libthai-devel-0.1.28-8.el9.x86_64.rpm 1.4 MB/s | 134 kB 00:00 2024-04-26T04:23:56.559 INFO:tasks.workunit.client.0.smithi098.stdout:(60/85): libtiff-devel-4.4.0-12.el9.x86_64.rpm 6.5 MB/s | 556 kB 00:00 2024-04-26T04:23:56.676 INFO:tasks.workunit.client.0.smithi098.stdout:(61/85): libxcb-devel-1.13.1-9.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-04-26T04:23:56.701 INFO:tasks.workunit.client.0.smithi098.stdout:(62/85): libxkbcommon-devel-1.0.3-4.el9.x86_64. 353 kB/s | 62 kB 00:00 2024-04-26T04:23:56.726 INFO:tasks.workunit.client.0.smithi098.stdout:(63/85): libxkbfile-1.1.0-8.el9.x86_64.rpm 537 kB/s | 89 kB 00:00 2024-04-26T04:23:56.810 INFO:tasks.workunit.client.0.smithi098.stdout:(64/85): pango-devel-1.48.7-3.el9.x86_64.rpm 1.7 MB/s | 149 kB 00:00 2024-04-26T04:23:56.836 INFO:tasks.workunit.client.0.smithi098.stdout:(65/85): pcre-cpp-8.44-3.el9.3.x86_64.rpm 247 kB/s | 27 kB 00:00 2024-04-26T04:23:56.869 INFO:tasks.workunit.client.0.smithi098.stdout:(66/85): libxml2-devel-2.9.13-5.el9.x86_64.rpm 6.2 MB/s | 898 kB 00:00 2024-04-26T04:23:56.894 INFO:tasks.workunit.client.0.smithi098.stdout:(67/85): pcre-devel-8.44-3.el9.3.x86_64.rpm 5.9 MB/s | 507 kB 00:00 2024-04-26T04:23:56.928 INFO:tasks.workunit.client.0.smithi098.stdout:(68/85): pcre-utf16-8.44-3.el9.3.x86_64.rpm 2.0 MB/s | 185 kB 00:00 2024-04-26T04:23:56.962 INFO:tasks.workunit.client.0.smithi098.stdout:(69/85): pcre-utf32-8.44-3.el9.3.x86_64.rpm 1.9 MB/s | 176 kB 00:00 2024-04-26T04:23:56.987 INFO:tasks.workunit.client.0.smithi098.stdout:(70/85): pcre2-devel-10.40-5.el9.x86_64.rpm 5.5 MB/s | 512 kB 00:00 2024-04-26T04:23:57.012 INFO:tasks.workunit.client.0.smithi098.stdout:(71/85): pcre2-utf16-10.40-5.el9.x86_64.rpm 2.5 MB/s | 214 kB 00:00 2024-04-26T04:23:57.038 INFO:tasks.workunit.client.0.smithi098.stdout:(72/85): pcre2-utf32-10.40-5.el9.x86_64.rpm 2.6 MB/s | 203 kB 00:00 2024-04-26T04:23:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:56 smithi098 ceph-mon[27184]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:57.063 INFO:tasks.workunit.client.0.smithi098.stdout:(73/85): pixman-devel-0.40.0-6.el9.x86_64.rpm 228 kB/s | 17 kB 00:00 2024-04-26T04:23:57.089 INFO:tasks.workunit.client.0.smithi098.stdout:(74/85): sysprof-capture-devel-3.40.1-3.el9.x86 792 kB/s | 60 kB 00:00 2024-04-26T04:23:57.114 INFO:tasks.workunit.client.0.smithi098.stdout:(75/85): wayland-devel-1.21.0-1.el9.x86_64.rpm 1.9 MB/s | 145 kB 00:00 2024-04-26T04:23:57.139 INFO:tasks.workunit.client.0.smithi098.stdout:(76/85): xkbcomp-1.4.4-4.el9.x86_64.rpm 1.3 MB/s | 100 kB 00:00 2024-04-26T04:23:57.164 INFO:tasks.workunit.client.0.smithi098.stdout:(77/85): xorg-x11-proto-devel-2022.2-1.el9.noar 4.0 MB/s | 309 kB 00:00 2024-04-26T04:23:57.206 INFO:tasks.workunit.client.0.smithi098.stdout:(78/85): xorg-x11-server-Xvfb-1.20.11-24.el9.x8 9.5 MB/s | 894 kB 00:00 2024-04-26T04:23:57.232 INFO:tasks.workunit.client.0.smithi098.stdout:(79/85): xorg-x11-server-common-1.20.11-24.el9. 390 kB/s | 36 kB 00:00 2024-04-26T04:23:57.257 INFO:tasks.workunit.client.0.smithi098.stdout:(80/85): xorg-x11-xauth-1.1-10.el9.x86_64.rpm 404 kB/s | 37 kB 00:00 2024-04-26T04:23:57.282 INFO:tasks.workunit.client.0.smithi098.stdout:(81/85): xz-devel-5.2.5-8.el9.x86_64.rpm 734 kB/s | 55 kB 00:00 2024-04-26T04:23:57.308 INFO:tasks.workunit.client.0.smithi098.stdout:(82/85): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-04-26T04:23:57.358 INFO:tasks.workunit.client.0.smithi098.stdout:(83/85): nss-3.90.0-7.el9.x86_64.rpm 9.2 MB/s | 706 kB 00:00 2024-04-26T04:23:57.383 INFO:tasks.workunit.client.0.smithi098.stdout:(84/85): nss-sysinit-3.90.0-7.el9.x86_64.rpm 265 kB/s | 20 kB 00:00 2024-04-26T04:23:57.564 INFO:tasks.workunit.client.0.smithi098.stdout:(85/85): GConf2-3.2.6-34.el9.x86_64.rpm 3.1 MB/s | 979 kB 00:00 2024-04-26T04:23:57.567 INFO:tasks.workunit.client.0.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-26T04:23:57.567 INFO:tasks.workunit.client.0.smithi098.stdout:Total 7.2 MB/s | 27 MB 00:03 2024-04-26T04:23:58.624 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction check 2024-04-26T04:23:58.700 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction check succeeded. 2024-04-26T04:23:58.700 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction test 2024-04-26T04:23:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:23:58 smithi098 ceph-mon[27184]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:23:59.142 INFO:tasks.workunit.client.0.smithi098.stdout:Transaction test succeeded. 2024-04-26T04:23:59.142 INFO:tasks.workunit.client.0.smithi098.stdout:Running transaction 2024-04-26T04:24:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:00 smithi098 ceph-mon[27184]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:01.987 INFO:tasks.workunit.client.0.smithi098.stdout: Preparing : 1/1 2024-04-26T04:24:02.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:01 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:24:02.056 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xorg-x11-proto-devel-2022.2-1.el9.noarch 1/87 2024-04-26T04:24:02.089 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 2/87 2024-04-26T04:24:02.114 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libpng-devel-2:1.6.37-12.el9.x86_64 3/87 2024-04-26T04:24:02.257 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libcanberra-gtk2-0.30-27.el9.x86_64 4/87 2024-04-26T04:24:02.306 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gtk2-2.24.33-8.el9.x86_64 5/87 2024-04-26T04:24:02.339 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libffi-devel-3.4.2-8.el9.x86_64 6/87 2024-04-26T04:24:02.370 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libICE-1.0.10-8.el9.x86_64 7/87 2024-04-26T04:24:02.402 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : fribidi-devel-1.0.10-6.el9.2.x86_64 8/87 2024-04-26T04:24:02.441 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : cmake-filesystem-3.26.5-2.el9.x86_64 9/87 2024-04-26T04:24:02.467 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : dbus-devel-1:1.12.20-8.el9.x86_64 10/87 2024-04-26T04:24:02.493 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libSM-1.2.3-10.el9.x86_64 11/87 2024-04-26T04:24:02.528 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXt-1.2.0-6.el9.x86_64 12/87 2024-04-26T04:24:02.560 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXmu-1.1.3-8.el9.x86_64 13/87 2024-04-26T04:24:02.681 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xorg-x11-xauth-1:1.1-10.el9.x86_64 14/87 2024-04-26T04:24:02.723 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : wayland-devel-1.21.0-1.el9.x86_64 15/87 2024-04-26T04:24:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:02 smithi098 ceph-mon[27184]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:24:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:24:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:24:03.712 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXau-devel-1.0.9-8.el9.x86_64 16/87 2024-04-26T04:24:04.093 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxcb-devel-1.13.1-9.el9.x86_64 17/87 2024-04-26T04:24:04.160 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libX11-devel-1.7.0-9.el9.x86_64 18/87 2024-04-26T04:24:04.191 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXext-devel-1.3.4-8.el9.x86_64 19/87 2024-04-26T04:24:04.225 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXfixes-devel-5.0.3-16.el9.x86_64 20/87 2024-04-26T04:24:04.279 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXrender-devel-0.9.10-16.el9.x86_64 21/87 2024-04-26T04:24:04.323 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXi-devel-1.7.10-8.el9.x86_64 22/87 2024-04-26T04:24:04.359 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXcursor-devel-1.2.0-7.el9.x86_64 23/87 2024-04-26T04:24:04.393 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXrandr-devel-1.5.2-8.el9.x86_64 24/87 2024-04-26T04:24:04.424 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXcomposite-devel-0.4.5-7.el9.x86_64 25/87 2024-04-26T04:24:04.453 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXinerama-devel-1.1.4-10.el9.x86_64 26/87 2024-04-26T04:24:04.482 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXtst-devel-1.2.3-16.el9.x86_64 27/87 2024-04-26T04:24:04.512 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXdamage-devel-1.1.5-7.el9.x86_64 28/87 2024-04-26T04:24:04.569 INFO:tasks.workunit.client.0.smithi098.stdout: Upgrading : nss-sysinit-3.90.0-7.el9.x86_64 29/87 2024-04-26T04:24:04.641 INFO:tasks.workunit.client.0.smithi098.stdout: Upgrading : nss-3.90.0-7.el9.x86_64 30/87 2024-04-26T04:24:04.659 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: nss-3.90.0-7.el9.x86_64 30/87 2024-04-26T04:24:04.857 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xz-devel-5.2.5-8.el9.x86_64 31/87 2024-04-26T04:24:04.891 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxml2-devel-2.9.13-5.el9.x86_64 32/87 2024-04-26T04:24:04.927 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxkbcommon-devel-1.0.3-4.el9.x86_64 33/87 2024-04-26T04:24:04.955 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : sysprof-capture-devel-3.40.1-3.el9.x86_64 34/87 2024-04-26T04:24:04.982 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pixman-devel-0.40.0-6.el9.x86_64 35/87 2024-04-26T04:24:05.008 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre2-utf32-10.40-5.el9.x86_64 36/87 2024-04-26T04:24:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:04 smithi098 ceph-mon[27184]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:05.119 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre2-utf16-10.40-5.el9.x86_64 37/87 2024-04-26T04:24:05.163 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre2-devel-10.40-5.el9.x86_64 38/87 2024-04-26T04:24:05.196 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre-utf32-8.44-3.el9.3.x86_64 39/87 2024-04-26T04:24:05.227 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre-utf16-8.44-3.el9.3.x86_64 40/87 2024-04-26T04:24:05.314 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre-cpp-8.44-3.el9.3.x86_64 41/87 2024-04-26T04:24:05.350 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pcre-devel-8.44-3.el9.3.x86_64 42/87 2024-04-26T04:24:05.373 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libxkbfile-1.1.0-8.el9.x86_64 43/87 2024-04-26T04:24:05.401 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xkbcomp-1.4.4-4.el9.x86_64 44/87 2024-04-26T04:24:05.522 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xorg-x11-server-common-1.20.11-24.el9.x86_64 45/87 2024-04-26T04:24:05.568 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libtiff-devel-4.4.0-12.el9.x86_64 46/87 2024-04-26T04:24:05.678 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libsepol-devel-3.6-1.el9.x86_64 47/87 2024-04-26T04:24:05.937 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libselinux-devel-3.6-1.el9.x86_64 48/87 2024-04-26T04:24:05.982 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libicu-devel-67.1-9.el9.x86_64 49/87 2024-04-26T04:24:06.017 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-opengl-1:1.3.4-1.el9.x86_64 50/87 2024-04-26T04:24:06.042 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-gles-1:1.3.4-1.el9.x86_64 51/87 2024-04-26T04:24:06.084 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-core-devel-1:1.3.4-1.el9.x86_64 52/87 2024-04-26T04:24:06.117 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libglvnd-devel-1:1.3.4-1.el9.x86_64 53/87 2024-04-26T04:24:06.144 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libepoxy-devel-1.5.5-4.el9.x86_64 54/87 2024-04-26T04:24:06.171 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libfontenc-1.1.3-17.el9.x86_64 55/87 2024-04-26T04:24:06.226 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXfont2-2.0.3-12.el9.x86_64 56/87 2024-04-26T04:24:06.301 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libdatrie-devel-0.2.13-4.el9.x86_64 57/87 2024-04-26T04:24:06.330 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libthai-devel-0.1.28-8.el9.x86_64 58/87 2024-04-26T04:24:06.354 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libblkid-devel-2.37.4-18.el9.x86_64 59/87 2024-04-26T04:24:06.525 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libmount-devel-2.37.4-18.el9.x86_64 60/87 2024-04-26T04:24:06.609 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : glib2-devel-2.68.4-14.el9.x86_64 61/87 2024-04-26T04:24:06.643 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : atk-devel-2.36.0-5.el9.x86_64 62/87 2024-04-26T04:24:06.703 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gdk-pixbuf2-devel-2.42.6-3.el9.x86_64 63/87 2024-04-26T04:24:06.735 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : at-spi2-core-devel-2.40.3-1.el9.x86_64 64/87 2024-04-26T04:24:06.762 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : at-spi2-atk-devel-2.38.0-4.el9.x86_64 65/87 2024-04-26T04:24:06.786 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXdmcp-1.1.3-8.el9.x86_64 66/87 2024-04-26T04:24:06.814 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : harfbuzz-icu-2.7.4-10.el9.x86_64 67/87 2024-04-26T04:24:06.840 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : graphite2-devel-1.3.14-9.el9.x86_64 68/87 2024-04-26T04:24:06.873 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : dbus-glib-0.110-13.el9.x86_64 69/87 2024-04-26T04:24:06.900 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : bzip2-devel-1.0.8-8.el9.x86_64 70/87 2024-04-26T04:24:06.934 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : brotli-1.0.9-6.el9.x86_64 71/87 2024-04-26T04:24:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:06 smithi098 ceph-mon[27184]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:07.057 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : brotli-devel-1.0.9-6.el9.x86_64 72/87 2024-04-26T04:24:07.219 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : harfbuzz-devel-2.7.4-10.el9.x86_64 73/87 2024-04-26T04:24:07.344 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : freetype-devel-2.10.4-9.el9.x86_64 74/87 2024-04-26T04:24:07.425 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : fontconfig-devel-2.14.0-2.el9.x86_64 75/87 2024-04-26T04:24:07.454 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : cairo-devel-1.17.4-7.el9.x86_64 76/87 2024-04-26T04:24:07.481 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : cairo-gobject-devel-1.17.4-7.el9.x86_64 77/87 2024-04-26T04:24:07.528 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXft-devel-2.3.3-8.el9.x86_64 78/87 2024-04-26T04:24:07.820 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : pango-devel-1.48.7-3.el9.x86_64 79/87 2024-04-26T04:24:08.222 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gtk2-devel-2.24.33-8.el9.x86_64 80/87 2024-04-26T04:24:08.348 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : gtk3-devel-3.24.31-2.el9.x86_64 81/87 2024-04-26T04:24:08.397 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : GConf2-3.2.6-34.el9.x86_64 82/87 2024-04-26T04:24:08.416 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: GConf2-3.2.6-34.el9.x86_64 82/87 2024-04-26T04:24:08.451 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : xorg-x11-server-Xvfb-1.20.11-24.el9.x86_64 83/87 2024-04-26T04:24:08.476 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libnotify-devel-0.7.9-8.el9.x86_64 84/87 2024-04-26T04:24:08.500 INFO:tasks.workunit.client.0.smithi098.stdout: Installing : libXScrnSaver-1.2.3-10.el9.x86_64 85/87 2024-04-26T04:24:08.525 INFO:tasks.workunit.client.0.smithi098.stdout: Cleanup : nss-3.90.0-6.el9.x86_64 86/87 2024-04-26T04:24:08.555 INFO:tasks.workunit.client.0.smithi098.stdout: Cleanup : nss-sysinit-3.90.0-6.el9.x86_64 87/87 2024-04-26T04:24:08.900 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: nss-3.90.0-7.el9.x86_64 87/87 2024-04-26T04:24:09.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:08 smithi098 ceph-mon[27184]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:10 smithi098 ceph-mon[27184]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:11.878 INFO:tasks.workunit.client.0.smithi098.stdout: Running scriptlet: nss-sysinit-3.90.0-6.el9.x86_64 87/87 2024-04-26T04:24:11.878 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : at-spi2-atk-devel-2.38.0-4.el9.x86_64 1/87 2024-04-26T04:24:11.878 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : at-spi2-core-devel-2.40.3-1.el9.x86_64 2/87 2024-04-26T04:24:11.878 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : atk-devel-2.36.0-5.el9.x86_64 3/87 2024-04-26T04:24:11.878 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : brotli-1.0.9-6.el9.x86_64 4/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : brotli-devel-1.0.9-6.el9.x86_64 5/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : bzip2-devel-1.0.8-8.el9.x86_64 6/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : cairo-devel-1.17.4-7.el9.x86_64 7/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : cairo-gobject-devel-1.17.4-7.el9.x86_64 8/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : cmake-filesystem-3.26.5-2.el9.x86_64 9/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : dbus-devel-1:1.12.20-8.el9.x86_64 10/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : dbus-glib-0.110-13.el9.x86_64 11/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : fontconfig-devel-2.14.0-2.el9.x86_64 12/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : freetype-devel-2.10.4-9.el9.x86_64 13/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : fribidi-devel-1.0.10-6.el9.2.x86_64 14/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gdk-pixbuf2-devel-2.42.6-3.el9.x86_64 15/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : glib2-devel-2.68.4-14.el9.x86_64 16/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : graphite2-devel-1.3.14-9.el9.x86_64 17/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gtk2-2.24.33-8.el9.x86_64 18/87 2024-04-26T04:24:11.879 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gtk2-devel-2.24.33-8.el9.x86_64 19/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : gtk3-devel-3.24.31-2.el9.x86_64 20/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : harfbuzz-devel-2.7.4-10.el9.x86_64 21/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : harfbuzz-icu-2.7.4-10.el9.x86_64 22/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libICE-1.0.10-8.el9.x86_64 23/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libSM-1.2.3-10.el9.x86_64 24/87 2024-04-26T04:24:11.880 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libX11-devel-1.7.0-9.el9.x86_64 25/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXScrnSaver-1.2.3-10.el9.x86_64 26/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXau-devel-1.0.9-8.el9.x86_64 27/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXcomposite-devel-0.4.5-7.el9.x86_64 28/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXcursor-devel-1.2.0-7.el9.x86_64 29/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXdamage-devel-1.1.5-7.el9.x86_64 30/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXdmcp-1.1.3-8.el9.x86_64 31/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXext-devel-1.3.4-8.el9.x86_64 32/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXfixes-devel-5.0.3-16.el9.x86_64 33/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXfont2-2.0.3-12.el9.x86_64 34/87 2024-04-26T04:24:11.887 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXft-devel-2.3.3-8.el9.x86_64 35/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXi-devel-1.7.10-8.el9.x86_64 36/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXinerama-devel-1.1.4-10.el9.x86_64 37/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXmu-1.1.3-8.el9.x86_64 38/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXrandr-devel-1.5.2-8.el9.x86_64 39/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXrender-devel-0.9.10-16.el9.x86_64 40/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXt-1.2.0-6.el9.x86_64 41/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libXtst-devel-1.2.3-16.el9.x86_64 42/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libblkid-devel-2.37.4-18.el9.x86_64 43/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libcanberra-gtk2-0.30-27.el9.x86_64 44/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libdatrie-devel-0.2.13-4.el9.x86_64 45/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libepoxy-devel-1.5.5-4.el9.x86_64 46/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libffi-devel-3.4.2-8.el9.x86_64 47/87 2024-04-26T04:24:11.888 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libfontenc-1.1.3-17.el9.x86_64 48/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-core-devel-1:1.3.4-1.el9.x86_64 49/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-devel-1:1.3.4-1.el9.x86_64 50/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-gles-1:1.3.4-1.el9.x86_64 51/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libglvnd-opengl-1:1.3.4-1.el9.x86_64 52/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libicu-devel-67.1-9.el9.x86_64 53/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libmount-devel-2.37.4-18.el9.x86_64 54/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libnotify-devel-0.7.9-8.el9.x86_64 55/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libpng-devel-2:1.6.37-12.el9.x86_64 56/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libselinux-devel-3.6-1.el9.x86_64 57/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libsepol-devel-3.6-1.el9.x86_64 58/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libthai-devel-0.1.28-8.el9.x86_64 59/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libtiff-devel-4.4.0-12.el9.x86_64 60/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxcb-devel-1.13.1-9.el9.x86_64 61/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxkbcommon-devel-1.0.3-4.el9.x86_64 62/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxkbfile-1.1.0-8.el9.x86_64 63/87 2024-04-26T04:24:11.889 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : libxml2-devel-2.9.13-5.el9.x86_64 64/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pango-devel-1.48.7-3.el9.x86_64 65/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre-cpp-8.44-3.el9.3.x86_64 66/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre-devel-8.44-3.el9.3.x86_64 67/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre-utf16-8.44-3.el9.3.x86_64 68/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre-utf32-8.44-3.el9.3.x86_64 69/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre2-devel-10.40-5.el9.x86_64 70/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre2-utf16-10.40-5.el9.x86_64 71/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pcre2-utf32-10.40-5.el9.x86_64 72/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : pixman-devel-0.40.0-6.el9.x86_64 73/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : sysprof-capture-devel-3.40.1-3.el9.x86_64 74/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : wayland-devel-1.21.0-1.el9.x86_64 75/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xkbcomp-1.4.4-4.el9.x86_64 76/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xorg-x11-proto-devel-2022.2-1.el9.noarch 77/87 2024-04-26T04:24:11.890 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xorg-x11-server-Xvfb-1.20.11-24.el9.x86_64 78/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xorg-x11-server-common-1.20.11-24.el9.x86_64 79/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xorg-x11-xauth-1:1.1-10.el9.x86_64 80/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : xz-devel-5.2.5-8.el9.x86_64 81/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 82/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : GConf2-3.2.6-34.el9.x86_64 83/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nss-3.90.0-7.el9.x86_64 84/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nss-3.90.0-6.el9.x86_64 85/87 2024-04-26T04:24:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nss-sysinit-3.90.0-7.el9.x86_64 86/87 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: Verifying : nss-sysinit-3.90.0-6.el9.x86_64 87/87 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout:Upgraded: 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: nss-3.90.0-7.el9.x86_64 nss-sysinit-3.90.0-7.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout:Installed: 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: GConf2-3.2.6-34.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-atk-devel-2.38.0-4.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: at-spi2-core-devel-2.40.3-1.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: atk-devel-2.36.0-5.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: brotli-1.0.9-6.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: brotli-devel-1.0.9-6.el9.x86_64 2024-04-26T04:24:12.894 INFO:tasks.workunit.client.0.smithi098.stdout: bzip2-devel-1.0.8-8.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: cairo-devel-1.17.4-7.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: cairo-gobject-devel-1.17.4-7.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: cmake-filesystem-3.26.5-2.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: dbus-devel-1:1.12.20-8.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: dbus-glib-0.110-13.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: fontconfig-devel-2.14.0-2.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: freetype-devel-2.10.4-9.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: fribidi-devel-1.0.10-6.el9.2.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: gdk-pixbuf2-devel-2.42.6-3.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: glib2-devel-2.68.4-14.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: graphite2-devel-1.3.14-9.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: gtk2-2.24.33-8.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: gtk2-devel-2.24.33-8.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: gtk3-devel-3.24.31-2.el9.x86_64 2024-04-26T04:24:12.895 INFO:tasks.workunit.client.0.smithi098.stdout: harfbuzz-devel-2.7.4-10.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: harfbuzz-icu-2.7.4-10.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libICE-1.0.10-8.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libSM-1.2.3-10.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libX11-devel-1.7.0-9.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXScrnSaver-1.2.3-10.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXau-devel-1.0.9-8.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXcomposite-devel-0.4.5-7.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXcursor-devel-1.2.0-7.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXdamage-devel-1.1.5-7.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXdmcp-1.1.3-8.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXext-devel-1.3.4-8.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXfixes-devel-5.0.3-16.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXfont2-2.0.3-12.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXft-devel-2.3.3-8.el9.x86_64 2024-04-26T04:24:12.896 INFO:tasks.workunit.client.0.smithi098.stdout: libXi-devel-1.7.10-8.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXinerama-devel-1.1.4-10.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXmu-1.1.3-8.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXrandr-devel-1.5.2-8.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXrender-devel-0.9.10-16.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXt-1.2.0-6.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libXtst-devel-1.2.3-16.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libblkid-devel-2.37.4-18.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libcanberra-gtk2-0.30-27.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libdatrie-devel-0.2.13-4.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libepoxy-devel-1.5.5-4.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libffi-devel-3.4.2-8.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libfontenc-1.1.3-17.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-core-devel-1:1.3.4-1.el9.x86_64 2024-04-26T04:24:12.897 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-devel-1:1.3.4-1.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-gles-1:1.3.4-1.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libglvnd-opengl-1:1.3.4-1.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libicu-devel-67.1-9.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libmount-devel-2.37.4-18.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libnotify-devel-0.7.9-8.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libpng-devel-2:1.6.37-12.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libselinux-devel-3.6-1.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libsepol-devel-3.6-1.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libthai-devel-0.1.28-8.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libtiff-devel-4.4.0-12.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libxcb-devel-1.13.1-9.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbcommon-devel-1.0.3-4.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libxkbfile-1.1.0-8.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: libxml2-devel-2.9.13-5.el9.x86_64 2024-04-26T04:24:12.898 INFO:tasks.workunit.client.0.smithi098.stdout: pango-devel-1.48.7-3.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-cpp-8.44-3.el9.3.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-devel-8.44-3.el9.3.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-utf16-8.44-3.el9.3.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre-utf32-8.44-3.el9.3.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-devel-10.40-5.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-utf16-10.40-5.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pcre2-utf32-10.40-5.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: pixman-devel-0.40.0-6.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: sysprof-capture-devel-3.40.1-3.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: wayland-devel-1.21.0-1.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: xkbcomp-1.4.4-4.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-proto-devel-2022.2-1.el9.noarch 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-server-Xvfb-1.20.11-24.el9.x86_64 2024-04-26T04:24:12.899 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-server-common-1.20.11-24.el9.x86_64 2024-04-26T04:24:12.900 INFO:tasks.workunit.client.0.smithi098.stdout: xorg-x11-xauth-1:1.1-10.el9.x86_64 2024-04-26T04:24:12.900 INFO:tasks.workunit.client.0.smithi098.stdout: xz-devel-5.2.5-8.el9.x86_64 2024-04-26T04:24:12.900 INFO:tasks.workunit.client.0.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T04:24:12.900 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:24:12.900 INFO:tasks.workunit.client.0.smithi098.stdout:Complete! 2024-04-26T04:24:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:12 smithi098 ceph-mon[27184]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:13.359 INFO:tasks.workunit.client.0.smithi098.stderr:++ ceph mgr services 2024-04-26T04:24:13.359 INFO:tasks.workunit.client.0.smithi098.stderr:++ jq -r .dashboard 2024-04-26T04:24:13.444 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2017395822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 msgr2=0x7fdc50101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:13.444 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2017395822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc50101710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009880 tx=0x7fdc4402be10 comp rx=0 tx=0).stop 2024-04-26T04:24:13.444 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2017395822 shutdown_connections 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2017395822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc50101710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2017395822 >> 172.21.15.98:0/2017395822 conn(0x7fdc500fc8a0 msgr2=0x7fdc500fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2017395822 shutdown_connections 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2017395822 wait complete. 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 Processor -- start 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.442+0000 7fdc56ec8640 1 -- start start 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc56ec8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc56ec8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501068e0 con 0x7fdc501012f0 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc54c64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc54c64640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34804/0 (socket says 172.21.15.98:34804) 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc54c64640 1 -- 172.21.15.98:0/2696145469 learned_addr learned my addr 172.21.15.98:0/2696145469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc54c64640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc50075910 con 0x7fdc501012f0 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc54c64640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009230 tx=0x7fdc440035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc44043020 con 0x7fdc501012f0 2024-04-26T04:24:13.445 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc50073af0 con 0x7fdc501012f0 2024-04-26T04:24:13.446 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdc44003c00 con 0x7fdc501012f0 2024-04-26T04:24:13.446 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.443+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc44034440 con 0x7fdc501012f0 2024-04-26T04:24:13.446 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc50074040 con 0x7fdc501012f0 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7fdc44046020 con 0x7fdc501012f0 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc4dffb640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc4dffb640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc4dffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.444+0000 7fdc4dffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.445+0000 7fdc4dffb640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 0x7fdc4003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.445+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7fdc4403a040 con 0x7fdc501012f0 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.445+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc50107060 con 0x7fdc501012f0 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.445+0000 7fdc4ffff640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 0x7fdc4003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:13.447 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.445+0000 7fdc4ffff640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 0x7fdc4003b670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0098c0 tx=0x7fdc3c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:13.450 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.450+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdc4404d020 con 0x7fdc501012f0 2024-04-26T04:24:13.684 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.683+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr services"} v 0) -- 0x7fdc5018f6c0 con 0x7fdc501012f0 2024-04-26T04:24:13.684 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.684+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr services"}]=0 v14) ==== 60+0+50 (secure 0 0 0) 0x7fdc4403d7b0 con 0x7fdc501012f0 2024-04-26T04:24:13.851 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.850+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr services"} v 0) -- 0x7fdc5004fd10 con 0x7fdc501012f0 2024-04-26T04:24:13.851 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.851+0000 7fdc4dffb640 1 -- 172.21.15.98:0/2696145469 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr services"}]=0 v14) ==== 60+0+50 (secure 0 0 0) 0x7fdc44047cb0 con 0x7fdc501012f0 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.853+0000 7fdc56ec8640 10 mgrc shutdown 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.853+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 msgr2=0x7fdc4003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.853+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 0x7fdc4003b670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c0098c0 tx=0x7fdc3c006db0 comp rx=0 tx=0).stop 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 msgr2=0x7fdc500753d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009230 tx=0x7fdc440035f0 comp rx=0 tx=0).stop 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 shutdown_connections 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7fdc400391b0 0x7fdc4003b670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 --2- 172.21.15.98:0/2696145469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc501012f0 0x7fdc500753d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 >> 172.21.15.98:0/2696145469 conn(0x7fdc500fc8a0 msgr2=0x7fdc500fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 shutdown_connections 2024-04-26T04:24:13.855 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:24:13.854+0000 7fdc56ec8640 1 -- 172.21.15.98:0/2696145469 wait complete. 2024-04-26T04:24:13.878 INFO:tasks.workunit.client.0.smithi098.stderr:+ CYPRESS_BASE_URL=https://172.21.15.98:8443/ 2024-04-26T04:24:13.878 INFO:tasks.workunit.client.0.smithi098.stderr:+ export CYPRESS_BASE_URL 2024-04-26T04:24:13.878 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/../../../src/pybind/mgr/dashboard/frontend 2024-04-26T04:24:13.878 INFO:tasks.workunit.client.0.smithi098.stderr:+ npm ci --unsafe-perm 2024-04-26T04:24:14.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:13 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2696145469' entity='client.admin' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T04:24:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:14 smithi098 ceph-mon[27184]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:14 smithi098 ceph-mon[27184]: from='client.? 172.21.15.98:0/2696145469' entity='client.admin' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T04:24:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:16 smithi098 ceph-mon[27184]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:18 smithi098 ceph-mon[27184]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:20 smithi098 ceph-mon[27184]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:22 smithi098 ceph-mon[27184]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:24 smithi098 ceph-mon[27184]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:26 smithi098 ceph-mon[27184]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:29.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:28 smithi098 ceph-mon[27184]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:31.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:30 smithi098 ceph-mon[27184]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:33.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:32 smithi098 ceph-mon[27184]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:35.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:34 smithi098 ceph-mon[27184]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:37.049 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated inside@1.0.0: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. 2024-04-26T04:24:37.120 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-nullish-coalescing-operator@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-nullish-coalescing-operator instead. 2024-04-26T04:24:37.121 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-dynamic-import@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-dynamic-import instead. 2024-04-26T04:24:37.171 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-numeric-separator@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-numeric-separator instead. 2024-04-26T04:24:37.173 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-export-namespace-from@7.18.9: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-export-namespace-from instead. 2024-04-26T04:24:37.187 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-json-strings@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-json-strings instead. 2024-04-26T04:24:37.211 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-unicode-property-regex@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-unicode-property-regex instead. 2024-04-26T04:24:37.216 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-class-properties@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-properties instead. 2024-04-26T04:24:37.248 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated 2024-04-26T04:24:37.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:36 smithi098 ceph-mon[27184]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:37.306 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated sourcemap-codec@1.4.8: Please use @jridgewell/sourcemap-codec instead 2024-04-26T04:24:37.354 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated har-validator@5.1.5: this library is no longer supported 2024-04-26T04:24:37.369 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated 2024-04-26T04:24:37.377 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142 2024-04-26T04:24:37.381 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated domexception@1.0.1: Use your platform's native DOMException instead 2024-04-26T04:24:37.415 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @npmcli/move-file@2.0.1: This functionality has been moved to @npmcli/fs 2024-04-26T04:24:37.448 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-class-static-block@7.21.0: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-static-block instead. 2024-04-26T04:24:37.456 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @stylelint/postcss-markdown@0.36.2: Use the original unforked package instead: postcss-markdown 2024-04-26T04:24:37.478 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-logical-assignment-operators@7.20.7: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-logical-assignment-operators instead. 2024-04-26T04:24:37.665 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated abab@2.0.6: Use your platform's native atob() and btoa() methods instead 2024-04-26T04:24:37.700 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-private-methods@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-private-methods instead. 2024-04-26T04:24:37.735 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated 2024-04-26T04:24:37.837 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-private-property-in-object@7.21.11: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-private-property-in-object instead. 2024-04-26T04:24:37.839 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-optional-catch-binding@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-optional-catch-binding instead. 2024-04-26T04:24:38.101 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated w3c-hr-time@1.0.2: Use your platform's native performance.now() and performance.timeOrigin. 2024-04-26T04:24:38.190 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-async-generator-functions@7.20.7: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-async-generator-functions instead. 2024-04-26T04:24:38.237 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated 2024-04-26T04:24:38.448 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated domexception@4.0.0: Use your platform's native DOMException instead 2024-04-26T04:24:38.518 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-optional-chaining@7.21.0: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-optional-chaining instead. 2024-04-26T04:24:39.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:38 smithi098 ceph-mon[27184]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:39.340 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @babel/plugin-proposal-object-rest-spread@7.20.7: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-object-rest-spread instead. 2024-04-26T04:24:39.344 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated debug@4.2.0: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797) 2024-04-26T04:24:39.824 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated dommatrix@1.0.3: dommatrix is no longer maintained. Please use @thednp/dommatrix. 2024-04-26T04:24:39.975 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 2024-04-26T04:24:40.067 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated @stylelint/postcss-css-in-js@0.37.3: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info. 2024-04-26T04:24:40.186 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 2024-04-26T04:24:41.249 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142 2024-04-26T04:24:41.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:40 smithi098 ceph-mon[27184]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:42.165 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated gherkin@5.1.0: This package is now published under @cucumber/gherkin 2024-04-26T04:24:42.521 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated cucumber-expressions@6.6.2: This package is now published under @cucumber/cucumber-expressions 2024-04-26T04:24:42.726 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated cucumber-expressions@5.0.18: This package is now published under @cucumber/cucumber-expressions 2024-04-26T04:24:43.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:42 smithi098 ceph-mon[27184]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:44.186 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated cucumber-messages@8.0.0: This package is now published under @cucumber/messages 2024-04-26T04:24:44.773 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated gherkin@9.0.0: This package is now published under @cucumber/gherkin 2024-04-26T04:24:45.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:44 smithi098 ceph-mon[27184]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:45.728 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated cucumber@4.2.1: Cucumber is publishing new releases under @cucumber/cucumber 2024-04-26T04:24:47.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:46 smithi098 ceph-mon[27184]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:49.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:48 smithi098 ceph-mon[27184]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:51.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:50 smithi098 ceph-mon[27184]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:52.218 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js. 2024-04-26T04:24:52.464 INFO:tasks.workunit.client.0.smithi098.stderr:npm WARN deprecated core-js@3.6.4: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js. 2024-04-26T04:24:53.092 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:52 smithi098 ceph-mon[27184]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:55.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:54 smithi098 ceph-mon[27184]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:57.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:56 smithi098 ceph-mon[27184]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:24:59.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:24:58 smithi098 ceph-mon[27184]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:01.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:00 smithi098 ceph-mon[27184]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:02.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:01 smithi098 ceph-mon[27184]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:25:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:25:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:25:03.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:04.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:03 smithi098 ceph-mon[27184]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:06 smithi098 ceph-mon[27184]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:08 smithi098 ceph-mon[27184]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:10 smithi098 ceph-mon[27184]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:12 smithi098 ceph-mon[27184]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:14 smithi098 ceph-mon[27184]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:16 smithi098 ceph-mon[27184]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:18 smithi098 ceph-mon[27184]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:20 smithi098 ceph-mon[27184]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:22 smithi098 ceph-mon[27184]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:24 smithi098 ceph-mon[27184]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:26.196 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:26.196 INFO:tasks.workunit.client.0.smithi098.stdout:> ceph-dashboard@0.0.0 postinstall 2024-04-26T04:25:26.196 INFO:tasks.workunit.client.0.smithi098.stdout:> ngcc --properties es2015 browser module main --async false --first-only --tsconfig 'tsconfig.app.json' 2024-04-26T04:25:26.196 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:26 smithi098 ceph-mon[27184]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:27.659 INFO:tasks.workunit.client.0.smithi098.stdout:- @circlon/angular-tree-component [es2015/esm2015] (https://github.com/CirclonGroup/angular-tree-component.git) 2024-04-26T04:25:28.901 INFO:tasks.workunit.client.0.smithi098.stdout:- @swimlane/ngx-datatable [es2015/esm2015] (git+https://github.com/swimlane/ngx-datatable.git) 2024-04-26T04:25:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:28 smithi098 ceph-mon[27184]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:29.474 INFO:tasks.workunit.client.0.smithi098.stdout:- ng-block-ui [es2015/esm2015] (git+https://github.com/kuuurt13/ng-block-ui.git) 2024-04-26T04:25:29.615 INFO:tasks.workunit.client.0.smithi098.stdout:- ng-block-ui/http [es2015/esm2015] (git+https://github.com/kuuurt13/ng-block-ui.git) 2024-04-26T04:25:29.795 INFO:tasks.workunit.client.0.smithi098.stdout:- ng-block-ui/router [es2015/esm2015] (git+https://github.com/kuuurt13/ng-block-ui.git) 2024-04-26T04:25:29.907 INFO:tasks.workunit.client.0.smithi098.stdout:- ng-click-outside [module/esm5] (https://github.com/arkon/ng-click-outside.git) 2024-04-26T04:25:29.989 INFO:tasks.workunit.client.0.smithi098.stdout:- ngx-pipe-function [es2015/esm2015] (https://github.com/ArtemLanovyy/ngx-pipe-function) 2024-04-26T04:25:30.067 INFO:tasks.workunit.client.0.smithi098.stdout:- simplebar-angular [es2015/esm2015] () 2024-04-26T04:25:30.201 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:30.201 INFO:tasks.workunit.client.0.smithi098.stdout:added 2812 packages in 1m 2024-04-26T04:25:30.201 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:30.201 INFO:tasks.workunit.client.0.smithi098.stdout:279 packages are looking for funding 2024-04-26T04:25:30.201 INFO:tasks.workunit.client.0.smithi098.stdout: run `npm fund` for details 2024-04-26T04:25:30.202 INFO:tasks.workunit.client.0.smithi098.stderr:npm notice 2024-04-26T04:25:30.202 INFO:tasks.workunit.client.0.smithi098.stderr:npm notice New minor version of npm available! 10.2.4 -> 10.6.0 2024-04-26T04:25:30.202 INFO:tasks.workunit.client.0.smithi098.stderr:npm notice Changelog: 2024-04-26T04:25:30.202 INFO:tasks.workunit.client.0.smithi098.stderr:npm notice Run `npm install -g npm@10.6.0` to update! 2024-04-26T04:25:30.203 INFO:tasks.workunit.client.0.smithi098.stderr:npm notice 2024-04-26T04:25:30.228 INFO:tasks.workunit.client.0.smithi098.stderr:+ npx cypress verify 2024-04-26T04:25:30.910 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:30.913 INFO:tasks.workunit.client.0.smithi098.stdout:[STARTED] Task without title. 2024-04-26T04:25:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:30 smithi098 ceph-mon[27184]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:32.418 INFO:tasks.workunit.client.0.smithi098.stdout:[TITLE] Verified Cypress! /home/ubuntu/.cache/Cypress/12.17.4/Cypress 2024-04-26T04:25:32.418 INFO:tasks.workunit.client.0.smithi098.stdout:[SUCCESS] Verified Cypress! /home/ubuntu/.cache/Cypress/12.17.4/Cypress 2024-04-26T04:25:32.432 INFO:tasks.workunit.client.0.smithi098.stderr:+ npx cypress info 2024-04-26T04:25:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:32 smithi098 ceph-mon[27184]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:33.722 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:25:33.723 INFO:tasks.workunit.client.0.smithi098.stderr:DevTools listening on ws://127.0.0.1:35347/devtools/browser/4d477e7d-4e5a-46c4-b95e-cfd21af7ddfe 2024-04-26T04:25:33.740 INFO:tasks.workunit.client.0.smithi098.stderr:[80305:0426/042533.739962:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported. 2024-04-26T04:25:33.857 INFO:tasks.workunit.client.0.smithi098.stdout:Displaying Cypress info... 2024-04-26T04:25:33.858 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.861 INFO:tasks.workunit.client.0.smithi098.stdout:Detected 1 browser installed: 2024-04-26T04:25:33.861 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.862 INFO:tasks.workunit.client.0.smithi098.stdout:1. Chrome 2024-04-26T04:25:33.862 INFO:tasks.workunit.client.0.smithi098.stdout: - Name: chrome 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: - Channel: stable 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: - Version: 124.0.6367.91 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: - Executable: google-chrome 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout:Note: to run these browsers, pass : to the '--browser' field 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout:Examples: 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout:- cypress run --browser chrome 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.863 INFO:tasks.workunit.client.0.smithi098.stdout:Learn More: https://on.cypress.io/launching-browsers 2024-04-26T04:25:33.921 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.921 INFO:tasks.workunit.client.0.smithi098.stdout:Proxy Settings: none detected 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout:Environment Variables: 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout:CYPRESS_BASE_URL: https://172.21.15.98:8443/ 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout:Application Data: /home/ubuntu/.config/cypress/cy/development 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout:Browser Profiles: /home/ubuntu/.config/cypress/cy/development/browsers 2024-04-26T04:25:33.922 INFO:tasks.workunit.client.0.smithi098.stdout:Binary Caches: /home/ubuntu/.cache/Cypress 2024-04-26T04:25:33.923 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:33.925 INFO:tasks.workunit.client.0.smithi098.stdout:Cypress Version: 12.17.4 (stable) 2024-04-26T04:25:33.925 INFO:tasks.workunit.client.0.smithi098.stdout:System Platform: linux (CentOS - 9 2024-04-26T04:25:33.925 INFO:tasks.workunit.client.0.smithi098.stdout:) 2024-04-26T04:25:33.925 INFO:tasks.workunit.client.0.smithi098.stdout:System Memory: 33.3 GB free 31.6 GB 2024-04-26T04:25:33.937 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph orch device ls --refresh 2024-04-26T04:25:34.019 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- 172.21.15.98:0/2279875301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 msgr2=0x7f00801005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.019 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 --2- 172.21.15.98:0/2279875301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f00801005e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0078009c20 tx=0x7f007802c2e0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.019 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- 172.21.15.98:0/2279875301 shutdown_connections 2024-04-26T04:25:34.019 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 --2- 172.21.15.98:0/2279875301 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f00801005e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- 172.21.15.98:0/2279875301 >> 172.21.15.98:0/2279875301 conn(0x7f00800fb710 msgr2=0x7f00800fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- 172.21.15.98:0/2279875301 shutdown_connections 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- 172.21.15.98:0/2279875301 wait complete. 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 Processor -- start 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- start start 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0086dc9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00801057b0 con 0x7f00801001c0 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.016+0000 7f0084b65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0084b65640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40318/0 (socket says 172.21.15.98:40318) 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0084b65640 1 -- 172.21.15.98:0/3957751007 learned_addr learned my addr 172.21.15.98:0/3957751007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0084b65640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0080198dd0 con 0x7f00801001c0 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0084b65640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0078005cc0 tx=0x7f0078032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0078041020 con 0x7f00801001c0 2024-04-26T04:25:34.020 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0078003970 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0080198fa0 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0078039040 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.017+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0080199350 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f0078044020 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0080105f30 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 0x7f006c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f007806ef10 con 0x7f00801001c0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.018+0000 7f007ffff640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 0x7f006c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:34.021 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.019+0000 7f007ffff640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 0x7f006c03b6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00680098c0 tx=0x7f0068006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:34.024 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.023+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f007804b020 con 0x7f00801001c0 2024-04-26T04:25:34.209 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.208+0000 7f0086dc9640 20 mgrc start_command cmd: [{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:34.209 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.208+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}) -- 0x7f008002bfb0 con 0x7f006c039200 2024-04-26T04:25:34.211 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.210+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+399 (secure 0 0 0) 0x7f008002bfb0 con 0x7f006c039200 2024-04-26T04:25:34.211 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.210+0000 7f007dffb640 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:25:34.326 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.325+0000 7f0086dc9640 20 mgrc start_command cmd: [{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:34.327 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.325+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 1: {"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}) -- 0x7f008002c160 con 0x7f006c039200 2024-04-26T04:25:34.328 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.327+0000 7f007dffb640 1 -- 172.21.15.98:0/3957751007 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== mgr_command_reply(tid 1: 0 ) ==== 8+0+399 (secure 0 0 0) 0x7f008002c160 con 0x7f006c039200 2024-04-26T04:25:34.328 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.327+0000 7f007dffb640 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:25:34.328 INFO:tasks.workunit.client.0.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T04:25:34.328 INFO:tasks.workunit.client.0.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-26T04:25:34.329 INFO:tasks.workunit.client.0.smithi098.stdout:smithi165 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CE400BGN 372G No 7m ago Has a FileSystem, LVM detected 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 10 mgrc shutdown 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 msgr2=0x7f006c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 0x7f006c03b6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00680098c0 tx=0x7f0068006db0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 msgr2=0x7f008006a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0078005cc0 tx=0x7f0078032d40 comp rx=0 tx=0).stop 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 shutdown_connections 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f006c039200 0x7f006c03b6c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 --2- 172.21.15.98:0/3957751007 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00801001c0 0x7f008006a010 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 >> 172.21.15.98:0/3957751007 conn(0x7f00800fb710 msgr2=0x7f00800fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.329+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 shutdown_connections 2024-04-26T04:25:34.331 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.330+0000 7f0086dc9640 1 -- 172.21.15.98:0/3957751007 wait complete. 2024-04-26T04:25:34.340 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph orch ps --refresh 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 -- 172.21.15.98:0/477291792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 msgr2=0x7f4a8c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 --2- 172.21.15.98:0/477291792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c101710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009880 tx=0x7f4a7c02be10 comp rx=0 tx=0).stop 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 -- 172.21.15.98:0/477291792 shutdown_connections 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 --2- 172.21.15.98:0/477291792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c101710 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 -- 172.21.15.98:0/477291792 >> 172.21.15.98:0/477291792 conn(0x7f4a8c0fc8a0 msgr2=0x7f4a8c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 -- 172.21.15.98:0/477291792 shutdown_connections 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.421+0000 7f4a91274640 1 -- 172.21.15.98:0/477291792 wait complete. 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a91274640 1 Processor -- start 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a91274640 1 -- start start 2024-04-26T04:25:34.424 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a91274640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a91274640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c1068e0 con 0x7f4a8c1012f0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a8affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40336/0 (socket says 172.21.15.98:40336) 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a8affd640 1 -- 172.21.15.98:0/1019532657 learned_addr learned my addr 172.21.15.98:0/1019532657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.422+0000 7f4a8affd640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0758c0 con 0x7f4a8c1012f0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a8affd640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c02c800 tx=0x7f4a7c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c043020 con 0x7f4a8c1012f0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a7c03e450 con 0x7f4a8c1012f0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c034440 con 0x7f4a8c1012f0 2024-04-26T04:25:34.425 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c073af0 con 0x7f4a8c1012f0 2024-04-26T04:25:34.426 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.423+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c074040 con 0x7f4a8c1012f0 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c107060 con 0x7f4a8c1012f0 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f4a7c046020 con 0x7f4a8c1012f0 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 0x7f4a7803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f4a7c03a040 con 0x7f4a8c1012f0 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.424+0000 7f4a83fff640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 0x7f4a7803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:34.427 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.425+0000 7f4a83fff640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 0x7f4a7803b6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a740098c0 tx=0x7f4a74006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:34.430 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.429+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a7c04d020 con 0x7f4a8c1012f0 2024-04-26T04:25:34.622 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.621+0000 7f4a91274640 20 mgrc start_command cmd: [{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:34.622 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.621+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}) -- 0x7f4a8c18f6f0 con 0x7f4a78039200 2024-04-26T04:25:34.625 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.624+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+786 (secure 0 0 0) 0x7f4a8c18f6f0 con 0x7f4a78039200 2024-04-26T04:25:34.625 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.624+0000 7f4a88ff9640 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:25:34.740 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.739+0000 7f4a91274640 20 mgrc start_command cmd: [{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:34.741 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.739+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 1: {"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}) -- 0x7f4a8c04fd10 con 0x7f4a78039200 2024-04-26T04:25:34.744 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.744+0000 7f4a88ff9640 1 -- 172.21.15.98:0/1019532657 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== mgr_command_reply(tid 1: 0 ) ==== 8+0+786 (secure 0 0 0) 0x7f4a7401c330 con 0x7f4a78039200 2024-04-26T04:25:34.744 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.744+0000 7f4a88ff9640 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:mgr.a smithi098 *:9283,8765 running (8m) 5m ago 8m 484M - 19.0.0-2457-gddeaa917 b2d50899180c e8b8c17534f6 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:mon.a smithi098 running (8m) 5m ago 8m 37.1M 2048M 19.0.0-2457-gddeaa917 b2d50899180c a4e58ea6476d 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:osd.0 smithi098 running (6m) 5m ago 6m 32.3M 5696M 19.0.0-2457-gddeaa917 b2d50899180c fbbbc2a68d9d 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:osd.1 smithi098 running (6m) 5m ago 6m 41.7M 5696M 19.0.0-2457-gddeaa917 b2d50899180c aecb356b08b1 2024-04-26T04:25:34.745 INFO:tasks.workunit.client.0.smithi098.stdout:osd.2 smithi098 running (5m) 5m ago 5m 13.5M 5696M 19.0.0-2457-gddeaa917 b2d50899180c f62ef08a218d 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 10 mgrc shutdown 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 msgr2=0x7f4a7803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 0x7f4a7803b6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a740098c0 tx=0x7f4a74006db0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 msgr2=0x7f4a8c075380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c02c800 tx=0x7f4a7c003820 comp rx=0 tx=0).stop 2024-04-26T04:25:34.747 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 shutdown_connections 2024-04-26T04:25:34.748 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f4a78039200 0x7f4a7803b6c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.748 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 --2- 172.21.15.98:0/1019532657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c1012f0 0x7f4a8c075380 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:34.748 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 >> 172.21.15.98:0/1019532657 conn(0x7f4a8c0fc8a0 msgr2=0x7f4a8c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:34.748 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 shutdown_connections 2024-04-26T04:25:34.748 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:34.746+0000 7f4a91274640 1 -- 172.21.15.98:0/1019532657 wait complete. 2024-04-26T04:25:34.756 INFO:tasks.workunit.client.0.smithi098.stderr:+ sleep 10 2024-04-26T04:25:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:34 smithi098 ceph-mon[27184]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:34 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:25:36.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:36.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='client.14258 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='client.14260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:36.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:35 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:37.019 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:36 smithi098 ceph-mon[27184]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:38 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:25:39.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: Detected new or changed devices on smithi165 2024-04-26T04:25:39.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: Removing smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: Removing smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: Removing smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:39.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:39 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:40 smithi098 ceph-mon[27184]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:40 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:40 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:40 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:40 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:42 smithi098 ceph-mon[27184]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:44.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:43 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:44.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:43 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:44.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:43 smithi098 ceph-mon[27184]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:44.758 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph orch device ls --format=json 2024-04-26T04:25:44.758 INFO:tasks.workunit.client.0.smithi098.stderr:+ tee cypress/fixtures/orchestrator/inventory.json 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 -- 172.21.15.98:0/2879929699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 msgr2=0x7f38400ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 --2- 172.21.15.98:0/2879929699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f38400ff9e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3838009880 tx=0x7f383802be10 comp rx=0 tx=0).stop 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 -- 172.21.15.98:0/2879929699 shutdown_connections 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 --2- 172.21.15.98:0/2879929699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f38400ff9e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 -- 172.21.15.98:0/2879929699 >> 172.21.15.98:0/2879929699 conn(0x7f38400faef0 msgr2=0x7f38400fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 -- 172.21.15.98:0/2879929699 shutdown_connections 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.838+0000 7f38484ed640 1 -- 172.21.15.98:0/2879929699 wait complete. 2024-04-26T04:25:44.842 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 Processor -- start 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 -- start start 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38401068f0 con 0x7f38400ff5c0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f3846289640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f3846289640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35496/0 (socket says 172.21.15.98:35496) 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f3846289640 1 -- 172.21.15.98:0/3817925774 learned_addr learned my addr 172.21.15.98:0/3817925774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f3846289640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3840196560 con 0x7f38400ff5c0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f3846289640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3838009230 tx=0x7f38380035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3838043020 con 0x7f38400ff5c0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38401967f0 con 0x7f38400ff5c0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3838003c00 con 0x7f38400ff5c0 2024-04-26T04:25:44.843 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3838034440 con 0x7f38400ff5c0 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.839+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38401994d0 con 0x7f38400ff5c0 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f3838046020 con 0x7f38400ff5c0 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 0x7f382803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f383803a040 con 0x7f38400ff5c0 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f3845a88640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 0x7f382803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:44.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.840+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3840107070 con 0x7f38400ff5c0 2024-04-26T04:25:44.845 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.841+0000 7f3845a88640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 0x7f382803b670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38340098c0 tx=0x7f3834006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:44.846 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:44.846+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f383804d020 con 0x7f38400ff5c0 2024-04-26T04:25:45.038 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.037+0000 7f38484ed640 20 mgrc start_command cmd: [{"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:25:45.038 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.037+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f384018e550 con 0x7f38280391b0 2024-04-26T04:25:45.039 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.038+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+5186 (secure 0 0 0) 0x7f384018e550 con 0x7f38280391b0 2024-04-26T04:25:45.039 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.038+0000 7f383f7fe640 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:25:45.152 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.151+0000 7f38484ed640 20 mgrc start_command cmd: [{"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:25:45.152 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.151+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 1: {"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f384002bfb0 con 0x7f38280391b0 2024-04-26T04:25:45.153 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.152+0000 7f383f7fe640 1 -- 172.21.15.98:0/3817925774 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== mgr_command_reply(tid 1: 0 ) ==== 8+0+5186 (secure 0 0 0) 0x7f384002bfb0 con 0x7f38280391b0 2024-04-26T04:25:45.153 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.152+0000 7f383f7fe640 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:25:45.154 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:45.154 INFO:tasks.workunit.client.0.smithi098.stdout:[{"addr": "smithi098", "devices": [{"available": false, "ceph_device": true, "created": "2024-04-26T04:25:44.134566Z", "crush_device_class": null, "device_id": "INTEL_SSDPEDMD400G4_PHFT6204010H400BGN", "human_readable_type": "ssd", "lsm_data": {}, "lvs": [{"comment": "not used by ceph", "name": "lv_1"}, {"block_uuid": "x2Y3hO-J2UJ-a0ck-o1tX-29Ko-9kXV-pZCeLs", "cluster_fsid": "b735c86e-0383-11ef-bc93-c7b262605968", "cluster_name": "ceph", "name": "lv_2", "osd_fsid": "528099cf-045c-41a0-8f9d-1730e8f25ca4", "osd_id": "2", "osdspec_affinity": "None", "type": "block"}, {"block_uuid": "2fE1FV-1Z8j-CkQJ-D0Yv-IwdD-uHAp-SF8LJQ", "cluster_fsid": "b735c86e-0383-11ef-bc93-c7b262605968", "cluster_name": "ceph", "name": "lv_3", "osd_fsid": "6ae29254-9452-4eea-b467-2f4d92354b21", "osd_id": "1", "osdspec_affinity": "None", "type": "block"}, {"block_uuid": "iRjA4R-uGMo-HIpp-UGkc-8E2L-zl4l-Si1OBi", "cluster_fsid": "b735c86e-0383-11ef-bc93-c7b262605968", "cluster_name": "ceph", "name": "lv_4", "osd_fsid": "0aa5bce2-dff3-4c44-92cb-dcdb5f0d7be9", "osd_id": "0", "osdspec_affinity": "None", "type": "block"}, {"comment": "not used by ceph", "name": "lv_5"}], "path": "/dev/nvme0n1", "rejected_reasons": ["Has a FileSystem", "LVM detected"], "sys_api": {"actuators": null, "device_nodes": "nvme0n1", "devname": "nvme0n1", "human_readable_size": "372.61 GB", "id_bus": "", "model": "INTEL SSDPEDMD400G4", "nr_requests": "1023", "parent": "/dev/nvme0n1", "partitions": {}, "path": "/dev/nvme0n1", "removable": "0", "rev": "", "ro": "0", "rotational": "0", "sas_address": "", "sas_device_handle": "", "scheduler_mode": "none", "sectors": 0, "sectorsize": "512", "size": 400088457216.0, "support_discard": "512", "type": "disk", "vendor": ""}}], "labels": [], "name": "smithi098"}, {"addr": "smithi165", "devices": [{"available": false, "ceph_device": false, "created": "2024-04-26T04:25:42.992154Z", "crush_device_class": null, "device_id": "INTEL_SSDPEDMD400G4_CVFT623300CE400BGN", "human_readable_type": "ssd", "lsm_data": {}, "lvs": [{"comment": "not used by ceph", "name": "lv_1"}, {"comment": "not used by ceph", "name": "lv_2"}, {"comment": "not used by ceph", "name": "lv_3"}, {"comment": "not used by ceph", "name": "lv_4"}, {"comment": "not used by ceph", "name": "lv_5"}], "path": "/dev/nvme0n1", "rejected_reasons": ["Has a FileSystem", "LVM detected"], "sys_api": {"actuators": null, "device_nodes": "nvme0n1", "devname": "nvme0n1", "human_readable_size": "372.61 GB", "id_bus": "", "model": "INTEL SSDPEDMD400G4", "nr_requests": "1023", "parent": "/dev/nvme0n1", "partitions": {}, "path": "/dev/nvme0n1", "removable": "0", "rev": "", "ro": "0", "rotational": "0", "sas_address": "", "sas_device_handle": "", "scheduler_mode": "none", "sectors": 0, "sectorsize": "512", "size": 400088457216.0, "support_discard": "512", "type": "disk", "vendor": ""}}, {"available": true, "ceph_device": false, "created": "2024-04-26T04:25:42.992136Z", "crush_device_class": null, "device_id": "LIO-ORG_lun1_2422da9b-f5f4-417d-8f1b-cb40e1e42d6b", "human_readable_type": "hdd", "lsm_data": {}, "lvs": [], "path": "/dev/sdb", "rejected_reasons": [], "sys_api": {"actuators": null, "device_nodes": "sdb", "devname": "sdb", "human_readable_size": "15.00 GB", "id_bus": "scsi", "model": "lun1", "nr_requests": "226", "parent": "/dev/sdb", "partitions": {}, "path": "/dev/sdb", "removable": "0", "rev": "4.0", "ro": "0", "rotational": "1", "sas_address": "", "sas_device_handle": "", "scheduler_mode": "mq-deadline", "sectors": 0, "sectorsize": "512", "size": 16106127360.0, "support_discard": "512", "type": "disk", "vendor": "LIO-ORG"}}, {"available": true, "ceph_device": false, "created": "2024-04-26T04:25:42.992146Z", "crush_device_class": null, "device_id": "LIO-ORG_lun2_db001b09-3776-4c67-82e2-c3f9a39bb080", "human_readable_type": "hdd", "lsm_data": {}, "lvs": [], "path": "/dev/sdc", "rejected_reasons": [], "sys_api": {"actuators": null, "device_nodes": "sdc", "devname": "sdc", "human_readable_size": "15.00 GB", "id_bus": "scsi", "model": "lun2", "nr_requests": "226", "parent": "/dev/sdc", "partitions": {}, "path": "/dev/sdc", "removable": "0", "rev": "4.0", "ro": "0", "rotational": "1", "sas_address": "", "sas_device_handle": "", "scheduler_mode": "mq-deadline", "sectors": 0, "sectorsize": "512", "size": 16106127360.0, "support_discard": "512", "type": "disk", "vendor": "LIO-ORG"}}, {"available": true, "ceph_device": false, "created": "2024-04-26T04:25:42.992150Z", "crush_device_class": null, "device_id": "LIO-ORG_lun3_00d6c581-83d4-465c-b976-36b00876714d", "human_readable_type": "hdd", "lsm_data": {}, "lvs": [], "path": "/dev/sdd", "rejected_reasons": [], "sys_api": {"actuators": null, "device_nodes": "sdd", "devname": "sdd", "human_readable_size": "15.00 GB", "id_bus": "scsi", "model": "lun3", "nr_requests": "226", "parent": "/dev/sdd", "partitions": {}, "path": "/dev/sdd", "removable": "0", "rev": "4.0", "ro": "0", "rotational": "1", "sas_address": "", "sas_device_handle": "", "scheduler_mode": "mq-deadline", "sectors": 0, "sectorsize": "512", "size": 16106127360.0, "support_discard": "512", "type": "disk", "vendor": "LIO-ORG"}}], "labels": [], "name": "smithi165"}] 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 10 mgrc shutdown 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 msgr2=0x7f382803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 0x7f382803b670 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f38340098c0 tx=0x7f3834006db0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 msgr2=0x7f3840196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3838009230 tx=0x7f38380035f0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 shutdown_connections 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f38280391b0 0x7f382803b670 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 --2- 172.21.15.98:0/3817925774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38400ff5c0 0x7f3840196020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.154+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 >> 172.21.15.98:0/3817925774 conn(0x7f38400faef0 msgr2=0x7f38400fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.155+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 shutdown_connections 2024-04-26T04:25:45.156 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.155+0000 7f38484ed640 1 -- 172.21.15.98:0/3817925774 wait complete. 2024-04-26T04:25:45.165 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph orch ps --format=json 2024-04-26T04:25:45.166 INFO:tasks.workunit.client.0.smithi098.stderr:+ tee cypress/fixtures/orchestrator/services.json 2024-04-26T04:25:45.244 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 -- 172.21.15.98:0/600713002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 msgr2=0x7f9b281005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 --2- 172.21.15.98:0/600713002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281005f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009880 tx=0x7f9b2402be10 comp rx=0 tx=0).stop 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 -- 172.21.15.98:0/600713002 shutdown_connections 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 --2- 172.21.15.98:0/600713002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281005f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 -- 172.21.15.98:0/600713002 >> 172.21.15.98:0/600713002 conn(0x7f9b280fb760 msgr2=0x7f9b280fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 -- 172.21.15.98:0/600713002 shutdown_connections 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.242+0000 7f9b3073e640 1 -- 172.21.15.98:0/600713002 wait complete. 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b3073e640 1 Processor -- start 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b3073e640 1 -- start start 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b3073e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b3073e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b281057c0 con 0x7f9b281001d0 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b2e4da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b2e4da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35522/0 (socket says 172.21.15.98:35522) 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b2e4da640 1 -- 172.21.15.98:0/1909808127 learned_addr learned my addr 172.21.15.98:0/1909808127 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:25:45.245 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b2e4da640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b28196600 con 0x7f9b281001d0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b2e4da640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b2402c800 tx=0x7f9b240035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b24043020 con 0x7f9b281001d0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b28196890 con 0x7f9b281001d0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9b2403e450 con 0x7f9b281001d0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.243+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b24034440 con 0x7f9b281001d0 2024-04-26T04:25:45.246 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b28199570 con 0x7f9b281001d0 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f9b24046020 con 0x7f9b281001d0 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b1f7fe640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 0x7f9b1403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.244+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b28105f40 con 0x7f9b281001d0 2024-04-26T04:25:45.247 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.245+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f9b2403a040 con 0x7f9b281001d0 2024-04-26T04:25:45.248 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.245+0000 7f9b2dcd9640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 0x7f9b1403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:45.248 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.245+0000 7f9b2dcd9640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 0x7f9b1403b5b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b180098c0 tx=0x7f9b18006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:45.250 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.249+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b2404d020 con 0x7f9b281001d0 2024-04-26T04:25:45.435 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.434+0000 7f9b3073e640 20 mgrc start_command cmd: [{"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:25:45.435 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.434+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b2818e430 con 0x7f9b140390f0 2024-04-26T04:25:45.437 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.436+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4752 (secure 0 0 0) 0x7f9b2818e430 con 0x7f9b140390f0 2024-04-26T04:25:45.437 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.436+0000 7f9b1f7fe640 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:25:45.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:45.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:45.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:25:45.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: Updating smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: Updating smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:45.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:45 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:45.549 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.548+0000 7f9b3073e640 20 mgrc start_command cmd: [{"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}] 2024-04-26T04:25:45.549 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.548+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 1: {"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b2818e630 con 0x7f9b140390f0 2024-04-26T04:25:45.551 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.550+0000 7f9b1f7fe640 1 -- 172.21.15.98:0/1909808127 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== mgr_command_reply(tid 1: 0 ) ==== 8+0+4752 (secure 0 0 0) 0x7f9b1801e2a0 con 0x7f9b140390f0 2024-04-26T04:25:45.551 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.550+0000 7f9b1f7fe640 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:25:45.551 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:45.552 INFO:tasks.workunit.client.0.smithi098.stdout:[{"container_id": "e8b8c17534f6", "container_image_digests": ["quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4"], "container_image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448", "cpu_percentage": "7.92%", "created": "2024-04-26T04:17:07.926624Z", "daemon_id": "a", "daemon_name": "mgr.a", "daemon_type": "mgr", "events": ["2024-04-26T04:18:38.957653Z daemon:mgr.a [INFO] \"Reconfigured mgr.a on host 'smithi098'\""], "hostname": "smithi098", "is_active": true, "last_refresh": "2024-04-26T04:25:40.185191Z", "memory_usage": 513382809, "ports": [9283, 8765], "service_name": "mgr", "started": "2024-04-26T04:17:07.558797Z", "status": 1, "status_desc": "running", "version": "19.0.0-2457-gddeaa917"}, {"container_id": "a4e58ea6476d", "container_image_digests": ["quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4"], "container_image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448", "cpu_percentage": "0.81%", "created": "2024-04-26T04:17:00.921712Z", "daemon_id": "a", "daemon_name": "mon.a", "daemon_type": "mon", "events": ["2024-04-26T04:18:31.113219Z daemon:mon.a [INFO] \"Reconfigured mon.a on host 'smithi098'\""], "hostname": "smithi098", "is_active": false, "last_refresh": "2024-04-26T04:25:40.185089Z", "memory_request": 2147483648, "memory_usage": 42530242, "ports": [], "service_name": "mon", "started": "2024-04-26T04:17:05.542371Z", "status": 1, "status_desc": "running", "version": "19.0.0-2457-gddeaa917"}, {"container_id": "fbbbc2a68d9d", "container_image_digests": ["quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4"], "container_image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4", "cpu_percentage": "0.42%", "created": "2024-04-26T04:18:59.004262Z", "daemon_id": "0", "daemon_name": "osd.0", "daemon_type": "osd", "events": ["2024-04-26T04:18:59.052477Z daemon:osd.0 [INFO] \"Deployed osd.0 on host 'smithi098'\""], "hostname": "smithi098", "is_active": false, "last_refresh": "2024-04-26T04:25:40.185268Z", "memory_request": 5973003468, "memory_usage": 39720058, "ports": [], "service_name": "osd", "started": "2024-04-26T04:18:58.616640Z", "status": 1, "status_desc": "running", "version": "19.0.0-2457-gddeaa917"}, {"container_id": "aecb356b08b1", "container_image_digests": ["quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4"], "container_image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4", "cpu_percentage": "1.36%", "created": "2024-04-26T04:19:24.883946Z", "daemon_id": "1", "daemon_name": "osd.1", "daemon_type": "osd", "events": ["2024-04-26T04:19:24.931368Z daemon:osd.1 [INFO] \"Deployed osd.1 on host 'smithi098'\""], "hostname": "smithi098", "is_active": false, "last_refresh": "2024-04-26T04:25:40.185339Z", "memory_request": 5973003468, "memory_usage": 48454696, "ports": [], "service_name": "osd", "started": "2024-04-26T04:19:24.462160Z", "status": 1, "status_desc": "running", "version": "19.0.0-2457-gddeaa917"}, {"container_id": "f62ef08a218d", "container_image_digests": ["quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4"], "container_image_id": "b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4", "cpu_percentage": "1.43%", "created": "2024-04-26T04:19:51.198618Z", "daemon_id": "2", "daemon_name": "osd.2", "daemon_type": "osd", "events": ["2024-04-26T04:19:51.245577Z daemon:osd.2 [INFO] \"Deployed osd.2 on host 'smithi098'\""], "hostname": "smithi098", "is_active": false, "last_refresh": "2024-04-26T04:25:40.185421Z", "memory_request": 5973003468, "memory_usage": 49209671, "ports": [], "service_name": "osd", "started": "2024-04-26T04:19:50.825362Z", "status": 1, "status_desc": "running", "version": "19.0.0-2457-gddeaa917"}] 2024-04-26T04:25:45.553 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 10 mgrc shutdown 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 msgr2=0x7f9b1403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 0x7f9b1403b5b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b180098c0 tx=0x7f9b18006db0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 msgr2=0x7f9b281960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9b2402c800 tx=0x7f9b240035f0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 shutdown_connections 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f9b140390f0 0x7f9b1403b5b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 --2- 172.21.15.98:0/1909808127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b281001d0 0x7f9b281960c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 >> 172.21.15.98:0/1909808127 conn(0x7f9b280fb760 msgr2=0x7f9b280698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.552+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 shutdown_connections 2024-04-26T04:25:45.554 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.553+0000 7f9b3073e640 1 -- 172.21.15.98:0/1909808127 wait complete. 2024-04-26T04:25:45.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ DASHBOARD_ADMIN_SECRET_FILE=/tmp/dashboard-admin-secret.txt 2024-04-26T04:25:45.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ printf admin 2024-04-26T04:25:45.563 INFO:tasks.workunit.client.0.smithi098.stderr:+ ceph dashboard ac-user-set-password admin -i /tmp/dashboard-admin-secret.txt --force-password 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/1566616974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 msgr2=0x7f0cf0103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/1566616974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0103fd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009880 tx=0x7f0ce002be10 comp rx=0 tx=0).stop 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/1566616974 shutdown_connections 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/1566616974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0103fd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/1566616974 >> 172.21.15.98:0/1566616974 conn(0x7f0cf00fb530 msgr2=0x7f0cf00fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/1566616974 shutdown_connections 2024-04-26T04:25:45.646 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.643+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/1566616974 wait complete. 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cf4d9f640 1 Processor -- start 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cf4d9f640 1 -- start start 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cf4d9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cf4d9f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01066a0 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cef7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35540/0 (socket says 172.21.15.98:35540) 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.644+0000 7f0cef7fe640 1 -- 172.21.15.98:0/131851545 learned_addr learned my addr 172.21.15.98:0/131851545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0cef7fe640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf01963d0 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0cef7fe640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005ab0 tx=0x7f0ce00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ce0043020 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0ce003e450 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf0196660 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ce0034440 con 0x7f0cf0101be0 2024-04-26T04:25:45.647 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.645+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf0199340 con 0x7f0cf0101be0 2024-04-26T04:25:45.648 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 14) ==== 46399+0+0 (secure 0 0 0) 0x7f0ce0046020 con 0x7f0cf0101be0 2024-04-26T04:25:45.648 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 20 mgrc handle_mgr_map mgrmap(e 14) 2024-04-26T04:25:45.648 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 4 mgrc handle_mgr_map Got map version 14 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf0106e20 con 0x7f0cf0101be0 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 4 mgrc reconnect Starting new session with [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 0x7f0cdc03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3017+0+0 (secure 0 0 0) 0x7f0ce003a040 con 0x7f0cf0101be0 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ce6ffd640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 0x7f0cdc03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:25:45.649 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.646+0000 7f0ce6ffd640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 0x7f0cdc03b5b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80098c0 tx=0x7f0cd8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:25:45.651 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.651+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ce004d020 con 0x7f0cf0101be0 2024-04-26T04:25:45.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.843+0000 7f0cf4d9f640 20 mgrc start_command cmd: [{"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:45.844 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:45.843+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}) -- 0x7f0cf018fa40 con 0x7f0cdc0390f0 2024-04-26T04:25:46.061 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.060+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mgr.14150 v2:172.21.15.98:6800/2693888440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+253 (secure 0 0 0) 0x7f0cf018fa40 con 0x7f0cdc0390f0 2024-04-26T04:25:46.061 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.060+0000 7f0ced7fa640 20 mgrc handle_command_reply tid 0 r 0 2024-04-26T04:25:46.180 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.180+0000 7f0cf4d9f640 20 mgrc start_command cmd: [{"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}] 2024-04-26T04:25:46.181 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.180+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 --> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] -- mgr_command(tid 1: {"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}) -- 0x7f0cf018f480 con 0x7f0cdc0390f0 2024-04-26T04:25:46.398 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.397+0000 7f0ced7fa640 1 -- 172.21.15.98:0/131851545 <== mgr.14150 v2:172.21.15.98:6800/2693888440 2 ==== mgr_command_reply(tid 1: 0 ) ==== 8+0+253 (secure 0 0 0) 0x7f0cf018f480 con 0x7f0cdc0390f0 2024-04-26T04:25:46.398 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.397+0000 7f0ced7fa640 20 mgrc handle_command_reply tid 1 r 0 2024-04-26T04:25:46.398 INFO:tasks.workunit.client.0.smithi098.stdout:{"username": "admin", "password": "$2b$12$XcWrHShFAlplOVggbU2HZ.uGU78udPE0Nd38dJVRm4FD0gge2PHGy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714105546, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false} 2024-04-26T04:25:46.400 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 10 mgrc shutdown 2024-04-26T04:25:46.400 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 msgr2=0x7f0cdc03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 0x7f0cdc03b5b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80098c0 tx=0x7f0cd8006db0 comp rx=0 tx=0).stop 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 msgr2=0x7f0cf0195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005ab0 tx=0x7f0ce00035f0 comp rx=0 tx=0).stop 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 shutdown_connections 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:6800/2693888440,v1:172.21.15.98:6801/2693888440] conn(0x7f0cdc0390f0 0x7f0cdc03b5b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 --2- 172.21.15.98:0/131851545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cf0101be0 0x7f0cf0195e90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 >> 172.21.15.98:0/131851545 conn(0x7f0cf00fb530 msgr2=0x7f0cf00fbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.399+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 shutdown_connections 2024-04-26T04:25:46.401 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-26T04:25:46.400+0000 7f0cf4d9f640 1 -- 172.21.15.98:0/131851545 wait complete. 2024-04-26T04:25:46.410 INFO:tasks.workunit.client.0.smithi098.stderr:+ find cypress 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/rgw-status.json 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/orchestrator 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/orchestrator/inventory.json 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/orchestrator/services.json 2024-04-26T04:25:46.411 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/block-rbd-status.json 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/fixtures/nfs-ganesha-status.json 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/plugins 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/plugins/index.js 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/tsconfig.json 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/visualTests 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/visualTests/login.vrt-spec.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/visualTests/dashboard.vrt-spec.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/api-docs.po.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/user-mgmt.e2e-spec.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/notification.po.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/api-docs.e2e-spec.ts 2024-04-26T04:25:46.412 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/dashboard.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/dashboard.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/dashboard-v3.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/language.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/login.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/user-mgmt.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/role-mgmt.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/language.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/navigation.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/role-mgmt.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/notification.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/dashboard-v3.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/navigation.e2e-spec.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/ui/login.po.ts 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator 2024-04-26T04:25:46.413 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/grafana 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/grafana/grafana.feature 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/06-cluster-check.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/04-create-cluster-create-osds.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/07-osds.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/03-create-cluster-create-services.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/08-hosts.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/10-nfs-exports.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/05-create-cluster-review.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/01-create-cluster-welcome.feature 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/nfs 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/nfs/nfs-export.po.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/09-services.e2e-spec.ts 2024-04-26T04:25:46.414 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/workflow/02-create-cluster-add-host.feature 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/03-inventory.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/05-services.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/01-hosts.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/orchestrator/04-osds.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/monitors.po.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/logs.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/monitors.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/users.po.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/services.po.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/mgr-modules.po.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/hosts.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/hosts.po.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/configuration.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/crush-map.e2e-spec.ts 2024-04-26T04:25:46.415 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/create-cluster.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/crush-map.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/logs.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/osds.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/osds.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/inventory.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/users.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/configuration.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/cluster/mgr-modules.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/a11y 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/a11y/dashboard.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/a11y/navigation.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/pools 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/pools/pools.e2e-spec.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/pools/pools.po.ts 2024-04-26T04:25:46.416 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/page-helper.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/daemons.e2e-spec.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/daemons.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/users.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/buckets.e2e-spec.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/roles.e2e-spec.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/buckets.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/roles.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/rgw/users.e2e-spec.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/grafana.feature.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/global.feature.po.ts 2024-04-26T04:25:46.417 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/create-cluster 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/create-cluster/create-cluster.feature.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/forms-helper.feature.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/urls.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/common/table-helper.feature.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/filesystems 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/filesystems/snapshots.e2e-spec.feature 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/filesystems/subvolume-groups.e2e-spec.feature 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/filesystems/filesystems.e2e-spec.feature 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/filesystems/subvolumes.e2e-spec.feature 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/mirroring.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/images.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/iscsi.po.ts 2024-04-26T04:25:46.418 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/mirroring.e2e-spec.ts 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/images.e2e-spec.ts 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/e2e/block/iscsi.e2e-spec.ts 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/support 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/support/e2e.ts 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/support/commands.ts 2024-04-26T04:25:46.419 INFO:tasks.workunit.client.0.smithi098.stdout:cypress/support/eyes-index.d.ts 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ cypress_run cypress/e2e/orchestrator/01-hosts.e2e-spec.ts 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ local specs=cypress/e2e/orchestrator/01-hosts.e2e-spec.ts 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ local timeout= 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ local 'override_config=excludeSpecPattern=*.po.ts,retries=0,specPattern=cypress/e2e/orchestrator/01-hosts.e2e-spec.ts' 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ '[' x '!=' x ']' 2024-04-26T04:25:46.420 INFO:tasks.workunit.client.0.smithi098.stderr:+ npx cypress run --browser chrome --headless --config 'excludeSpecPattern=*.po.ts,retries=0,specPattern=cypress/e2e/orchestrator/01-hosts.e2e-spec.ts' 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:46.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:46 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:47.708 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-26T04:25:47.708 INFO:tasks.workunit.client.0.smithi098.stderr:DevTools listening on ws://127.0.0.1:41825/devtools/browser/af92003b-183f-4f86-8829-565b91511261 2024-04-26T04:25:47.724 INFO:tasks.workunit.client.0.smithi098.stderr:[86330:0426/042547.724074:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported. 2024-04-26T04:25:47.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:47 smithi098 ceph-mon[27184]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:47.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:47 smithi098 ceph-mon[27184]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-set-password", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T04:25:47.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:47 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:48.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:48 smithi098 ceph-mon[27184]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout:The experimentalSessionAndOrigin configuration option was removed in Cypress version 12.0.0. 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout:You can safely remove this option from your config. 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout:https://on.cypress.io/session 2024-04-26T04:25:48.997 INFO:tasks.workunit.client.0.smithi098.stdout:https://on.cypress.io/origin 2024-04-26T04:25:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:50 smithi098 ceph-mon[27184]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:51.228 INFO:tasks.workunit.client.0.smithi098.stderr:Couldn't determine Mocha version 2024-04-26T04:25:51.246 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.279 INFO:tasks.workunit.client.0.smithi098.stderr:tput: No value for $TERM and no -T specified 2024-04-26T04:25:51.280 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:25:51.280 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.280 INFO:tasks.workunit.client.0.smithi098.stdout: (Run Starting) 2024-04-26T04:25:51.280 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: │ Cypress: 12.17.4 │ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: │ Browser: Chrome 124 (headless) │ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: │ Node Version: v18.19.1 (/usr/bin/node) │ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: │ Specs: 1 found (01-hosts.e2e-spec.ts) │ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: │ Searched: cypress/e2e/orchestrator/01-hosts.e2e-spec.ts │ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: └────────────────────────────────────────────────────────────────────────────────────────────────┘ 2024-04-26T04:25:51.286 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.287 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.287 INFO:tasks.workunit.client.0.smithi098.stdout:──────────────────────────────────────────────────────────────────────────────────────────────────── 2024-04-26T04:25:51.288 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:51.288 INFO:tasks.workunit.client.0.smithi098.stdout: Running: 01-hosts.e2e-spec.ts (1 of 1) 2024-04-26T04:25:53.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:52 smithi098 ceph-mon[27184]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:54 smithi098 ceph-mon[27184]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:55.444 INFO:tasks.workunit.client.0.smithi098.stderr:Couldn't determine Mocha version 2024-04-26T04:25:55.451 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:55.458 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:25:55.594 INFO:tasks.workunit.client.0.smithi098.stdout: Hosts page 2024-04-26T04:25:55.594 INFO:tasks.workunit.client.0.smithi098.stdout: when Orchestrator is available 2024-04-26T04:25:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:56 smithi098 ceph-mon[27184]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:25:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:56 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:56 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:25:58.072 INFO:tasks.workunit.client.0.smithi098.stdout: ✓ should not add an existing host 2024-04-26T04:25:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:25:58 smithi098 ceph-mon[27184]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:00 smithi098 ceph-mon[27184]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: Added label _no_schedule to host smithi165 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: Added label _no_conf_keyring to host smithi165 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:03.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:02 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:04.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:04.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:26:04.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:26:04.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:04.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:04.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:26:05.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:05 smithi098 ceph-mon[27184]: Removing smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T04:26:05.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:05 smithi098 ceph-mon[27184]: Removing smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-26T04:26:05.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:05 smithi098 ceph-mon[27184]: Removing smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T04:26:06.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:06 smithi098 ceph-mon[27184]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:08 smithi098 ceph-mon[27184]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:10 smithi098 ceph-mon[27184]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:12 smithi098 ceph-mon[27184]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:14 smithi098 ceph-mon[27184]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:16 smithi098 ceph-mon[27184]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:18 smithi098 ceph-mon[27184]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:21.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:20 smithi098 ceph-mon[27184]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:22 smithi098 ceph-mon[27184]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:24 smithi098 ceph-mon[27184]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:26 smithi098 ceph-mon[27184]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:28 smithi098 ceph-mon[27184]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:30 smithi098 ceph-mon[27184]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:32 smithi098 ceph-mon[27184]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:34 smithi098 ceph-mon[27184]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:36 smithi098 ceph-mon[27184]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:38 smithi098 ceph-mon[27184]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:40 smithi098 ceph-mon[27184]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:42 smithi098 ceph-mon[27184]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:44 smithi098 ceph-mon[27184]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:46 smithi098 ceph-mon[27184]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:48 smithi098 ceph-mon[27184]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:50 smithi098 ceph-mon[27184]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:52.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:51 smithi098 ceph-mon[27184]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:54.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:54 smithi098 ceph-mon[27184]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:56 smithi098 ceph-mon[27184]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:26:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:26:58 smithi098 ceph-mon[27184]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:00 smithi098 ceph-mon[27184]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:02 smithi098 ceph-mon[27184]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:04 smithi098 ceph-mon[27184]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:04 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:27:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:27:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:27:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:27:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:06 smithi098 ceph-mon[27184]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:08 smithi098 ceph-mon[27184]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:11.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:10 smithi098 ceph-mon[27184]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:12 smithi098 ceph-mon[27184]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:14 smithi098 ceph-mon[27184]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:16 smithi098 ceph-mon[27184]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:18 smithi098 ceph-mon[27184]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:20 smithi098 ceph-mon[27184]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:22 smithi098 ceph-mon[27184]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:24 smithi098 ceph-mon[27184]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:26 smithi098 ceph-mon[27184]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:28 smithi098 ceph-mon[27184]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:31.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:30 smithi098 ceph-mon[27184]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:33.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:32 smithi098 ceph-mon[27184]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:35.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:34 smithi098 ceph-mon[27184]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:37.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:36 smithi098 ceph-mon[27184]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:39.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:38 smithi098 ceph-mon[27184]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:41.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:40 smithi098 ceph-mon[27184]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:43.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:42 smithi098 ceph-mon[27184]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:45.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:44 smithi098 ceph-mon[27184]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:47.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:46 smithi098 ceph-mon[27184]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:49.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:48 smithi098 ceph-mon[27184]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:51.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:50 smithi098 ceph-mon[27184]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:53.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:52 smithi098 ceph-mon[27184]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:55.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:54 smithi098 ceph-mon[27184]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:57.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:56 smithi098 ceph-mon[27184]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:27:59.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:27:58 smithi098 ceph-mon[27184]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:01.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:00 smithi098 ceph-mon[27184]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:03.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:02 smithi098 ceph-mon[27184]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:03.695 INFO:tasks.workunit.client.0.smithi098.stdout: 1) should drain and remove a host and then add it back 2024-04-26T04:28:05.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:04 smithi098 ceph-mon[27184]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:06.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:05 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:28:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:06 smithi098 ceph-mon[27184]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:06 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "device ls-by-host", "format": "json", "host": "smithi098"}]: dispatch 2024-04-26T04:28:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:06 smithi098 ceph-mon[27184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "device ls-by-host", "format": "json", "host": "smithi098"}]: dispatch 2024-04-26T04:28:07.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:06 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:07.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:06 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:07.311 INFO:tasks.workunit.client.0.smithi098.stdout: ✓ should display inventory 2024-04-26T04:28:08.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:07 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:28:08.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:07 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:28:08.043 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:07 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:08 smithi098 ceph-mon[27184]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:09.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:08 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "device ls-by-host", "format": "json", "host": "smithi098"}]: dispatch 2024-04-26T04:28:09.042 INFO:tasks.workunit.client.0.smithi098.stdout: ✓ should display daemons 2024-04-26T04:28:10.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:09 smithi098 ceph-mon[27184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "device ls-by-host", "format": "json", "host": "smithi098"}]: dispatch 2024-04-26T04:28:11.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:10 smithi098 ceph-mon[27184]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: Added label bar to host smithi098 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: Added label foo to host smithi098 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:28:12.791 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:28:12.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:28:12.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:12.792 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:12 smithi098 ceph-mon[27184]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T04:28:15.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T04:28:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T04:28:15.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:14 smithi098 ceph-mon[27184]: from='mgr.14150 172.21.15.98:0/1661116078' entity='mgr.a' 2024-04-26T04:28:16.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:15 smithi098 ceph-mon[27184]: Removed label bar from host smithi098 2024-04-26T04:28:16.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:15 smithi098 ceph-mon[27184]: Removed label foo from host smithi098 2024-04-26T04:28:16.810 INFO:tasks.workunit.client.0.smithi098.stdout: ✓ should edit host labels (6637ms) 2024-04-26T04:28:16.810 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: 4 passing (2m) 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: 1 failing 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: 1) Hosts page 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: when Orchestrator is available 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: should drain and remove a host and then add it back: 2024-04-26T04:28:16.837 INFO:tasks.workunit.client.0.smithi098.stdout: CypressError: Timed out retrying after 120000ms: `cy.parent()` failed because the page updated as a result of this command, but you tried to continue the command chain. The subject is no longer attached to the DOM, and Cypress cannot requery the page after commands such as `cy.parent()`. 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout:Common situations why this happens: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: - Your JS framework re-rendered asynchronously 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: - Your app code reacted to an event firing and removed the element 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout:You can typically solve this by breaking up a chain. For example, rewrite: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout:> `cy.get('button').click().should('have.class', 'active')` 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout:to 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.838 INFO:tasks.workunit.client.0.smithi098.stdout:> `cy.get('button').as('btn').click()` 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout:> `cy.get('@btn').should('have.class', 'active')` 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout:https://on.cypress.io/element-has-detached-from-dom 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at isAttached (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:144393:58) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at validateType (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:144263:9) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at Object.isType (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:144298:7) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at Array. (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:134720:24) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at $Cy.getSubjectFromChain (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:138069:34) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at $Cy.subject (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:138023:17) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at Object.subjectFn (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:143310:28) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at $Cy.verifyUpcomingAssertions (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:142658:31) 2024-04-26T04:28:16.839 INFO:tasks.workunit.client.0.smithi098.stdout: at onRetry (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:143305:15) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at tryCatcher (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1807:23) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise.attempt.Promise.try (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:4315:29) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at whenStable (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:143192:68) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:143133:14) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at tryCatcher (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1807:23) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise._settlePromiseFromHandler (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1519:31) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise._settlePromise (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1576:18) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise._settlePromise0 (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1621:10) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise._settlePromises (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1701:18) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at Promise._fulfill (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:1645:18) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at (https://172.21.15.98:8443/__cypress/runner/cypress_runner.js:5450:46) 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: From Your Spec Code: 2024-04-26T04:28:16.840 INFO:tasks.workunit.client.0.smithi098.stdout: at HostsPageHelper.checkLabelExists (cypress/e2e/cluster/hosts.po.ts:97:7) 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: at HostsPageHelper.drain (cypress/e2e/cluster/hosts.po.ts:166:9) 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: at descriptor.value [as drain] (cypress/e2e/page-helper.po.ts:25:11) 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: at Context.eval (cypress/e2e/orchestrator/01-hosts.e2e-spec.ts:27:12) 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:16.841 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:16 smithi098 ceph-mon[27184]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:17.852 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.852 INFO:tasks.workunit.client.0.smithi098.stdout: (Results) 2024-04-26T04:28:17.854 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.855 INFO:tasks.workunit.client.0.smithi098.stdout: ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ 2024-04-26T04:28:17.855 INFO:tasks.workunit.client.0.smithi098.stdout: │ Tests: 5 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Passing: 4 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Failing: 1 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Pending: 0 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Skipped: 0 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Screenshots: 1 │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Video: true │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Duration: 2 minutes, 21 seconds │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: │ Spec Ran: 01-hosts.e2e-spec.ts │ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: └────────────────────────────────────────────────────────────────────────────────────────────────┘ 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.856 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: (Screenshots) 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: - /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/s (1280x633) 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: creenshots/01-hosts.e2e-spec.ts/Hosts page -- when Orchestrator is available -- 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: should drain and remove a host and then add it back (failed).png 2024-04-26T04:28:17.857 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.859 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.859 INFO:tasks.workunit.client.0.smithi098.stdout: (Video) 2024-04-26T04:28:17.859 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:17.860 INFO:tasks.workunit.client.0.smithi098.stdout: - Started compressing: Compressing to 32 CRF 2024-04-26T04:28:19.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:18 smithi098 ceph-mon[27184]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:21.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:20 smithi098 ceph-mon[27184]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:23.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:22 smithi098 ceph-mon[27184]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:25.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:24 smithi098 ceph-mon[27184]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:27.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:26 smithi098 ceph-mon[27184]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:27.915 INFO:tasks.workunit.client.0.smithi098.stdout: Compression progress: 98% 2024-04-26T04:28:28.566 INFO:tasks.workunit.client.0.smithi098.stdout: - Finished compressing: 10 seconds 2024-04-26T04:28:28.569 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.569 INFO:tasks.workunit.client.0.smithi098.stdout: - Video output: /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr/dashboard/frontend/cypress/videos/01-hosts.e2e-spec.ts.mp4 2024-04-26T04:28:28.569 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.571 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.583 INFO:tasks.workunit.client.0.smithi098.stdout:================================================================================ 2024-04-26T04:28:28.583 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.584 INFO:tasks.workunit.client.0.smithi098.stderr:tput: No value for $TERM and no -T specified 2024-04-26T04:28:28.584 INFO:tasks.workunit.client.0.smithi098.stdout: (Run Finished) 2024-04-26T04:28:28.586 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.586 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: Spec Tests Passing Failing Pending Skipped 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: ┌────────────────────────────────────────────────────────────────────────────────────────────────┐ 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: │ ✖ 01-hosts.e2e-spec.ts 02:21 5 4 1 - - │ 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: └────────────────────────────────────────────────────────────────────────────────────────────────┘ 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: ✖ 1 of 1 failed (100%) 02:21 5 4 1 - - 2024-04-26T04:28:28.588 INFO:tasks.workunit.client.0.smithi098.stdout: 2024-04-26T04:28:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:28:28.703 INFO:tasks.workunit:Stopping ['cephadm/test_dashboard_e2e.sh'] on client.0... 2024-04-26T04:28:28.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T04:28:29.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:28 smithi098 ceph-mon[27184]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:30.877 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2024-04-26T04:28:31.119 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=96b4ef22a2f540d9b16de4ba220fc1ed Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2024-04-26T04:28:31.121 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T04:28:31.132 INFO:tasks.cephadm:Teardown begin 2024-04-26T04:28:31.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:28:31.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:30 smithi098 ceph-mon[27184]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-26T04:28:31.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:28:31.209 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T04:28:31.209 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T04:28:31.245 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T04:28:31.275 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T04:28:31.275 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T04:28:31.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a 2024-04-26T04:28:31.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:31 smithi098 systemd[1]: Stopping Ceph mon.a for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:28:31.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a[27180]: 2024-04-26T04:28:31.583+0000 7fd432a92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:28:31.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:31 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a[27180]: 2024-04-26T04:28:31.583+0000 7fd432a92700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T04:28:31.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:31 smithi098 podman[87912]: 2024-04-26 04:28:31.716245375 +0000 UTC m=+0.239051578 container died a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T04:28:32.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:32 smithi098 podman[87912]: 2024-04-26 04:28:32.09231705 +0000 UTC m=+0.615123264 container cleanup a4e58ea6476da5001352e9e3c2c59bfda91467b3a53f0514d350b278dccee8ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T04:28:32.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 26 04:28:32 smithi098 bash[87912]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-mon-a 2024-04-26T04:28:32.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T04:28:32.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T04:28:32.583 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T04:28:32.584 INFO:tasks.cephadm.mgr.a:Stopping mgr.a... 2024-04-26T04:28:32.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a 2024-04-26T04:28:32.881 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:32 smithi098 systemd[1]: Stopping Ceph mgr.a for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:28:33.157 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:28:32.880+0000 7f8343446700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:28:33.157 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:32 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a[27416]: 2024-04-26T04:28:32.880+0000 7f8343446700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T04:28:33.157 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:32 smithi098 podman[88042]: 2024-04-26 04:28:32.91088952 +0000 UTC m=+0.137259010 container died e8b8c17534f6a101945a05eaba39b0248721fd5941e24446a0cd3eca1d227379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T04:28:33.432 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 podman[88042]: 2024-04-26 04:28:33.230543534 +0000 UTC m=+0.456913016 container cleanup e8b8c17534f6a101945a05eaba39b0248721fd5941e24446a0cd3eca1d227379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-26T04:28:33.433 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 bash[88042]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a 2024-04-26T04:28:33.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service' 2024-04-26T04:28:33.686 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 podman[88056]: 2024-04-26 04:28:33.432099706 +0000 UTC m=+0.521399940 container remove e8b8c17534f6a101945a05eaba39b0248721fd5941e24446a0cd3eca1d227379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ddeaa91797339a19172c3036ff48cca58c12f448, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-mgr-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.build-date=20240424) 2024-04-26T04:28:33.687 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 systemd[1]: ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service: Deactivated successfully. 2024-04-26T04:28:33.687 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 systemd[1]: Stopped Ceph mgr.a for b735c86e-0383-11ef-bc93-c7b262605968. 2024-04-26T04:28:33.687 INFO:journalctl@ceph.mgr.a.smithi098.stdout:Apr 26 04:28:33 smithi098 systemd[1]: ceph-b735c86e-0383-11ef-bc93-c7b262605968@mgr.a.service: Consumed 48.631s CPU time. 2024-04-26T04:28:33.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T04:28:33.700 INFO:tasks.cephadm.mgr.a:Stopped mgr.a 2024-04-26T04:28:33.700 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T04:28:33.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.0 2024-04-26T04:28:34.042 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:33 smithi098 systemd[1]: Stopping Ceph osd.0 for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:28:34.541 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:34 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0[38585]: 2024-04-26T04:28:34.051+0000 7f1c92ef4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:28:34.541 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:34 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0[38585]: 2024-04-26T04:28:34.051+0000 7f1c92ef4700 -1 osd.0 21 *** Got signal Terminated *** 2024-04-26T04:28:34.542 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:34 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0[38585]: 2024-04-26T04:28:34.051+0000 7f1c92ef4700 -1 osd.0 21 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T04:28:39.637 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:39 smithi098 podman[88172]: 2024-04-26 04:28:39.363812807 +0000 UTC m=+5.435701704 container died fbbbc2a68d9d5114218ee4d5a468f5921d858cd28beff1169934fec5bead41ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T04:28:39.987 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:39 smithi098 podman[88172]: 2024-04-26 04:28:39.726838313 +0000 UTC m=+5.798727215 container cleanup fbbbc2a68d9d5114218ee4d5a468f5921d858cd28beff1169934fec5bead41ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T04:28:39.987 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:39 smithi098 bash[88172]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0 2024-04-26T04:28:40.291 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:39 smithi098 podman[88189]: 2024-04-26 04:28:39.986426189 +0000 UTC m=+0.617539723 container remove fbbbc2a68d9d5114218ee4d5a468f5921d858cd28beff1169934fec5bead41ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T04:28:40.599 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.246928771 +0000 UTC m=+0.017805625 image pull b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4 2024-04-26T04:28:40.599 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.395910611 +0000 UTC m=+0.166787496 container create 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T04:28:40.600 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.598726503 +0000 UTC m=+0.369603346 container init 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T04:28:40.902 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.602086196 +0000 UTC m=+0.372963048 container start 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-26T04:28:40.903 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.682094477 +0000 UTC m=+0.452971328 container attach 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T04:28:41.185 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:40 smithi098 podman[88282]: 2024-04-26 04:28:40.899032162 +0000 UTC m=+0.669909011 container died 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, io.buildah.version=1.33.5, ceph=True) 2024-04-26T04:28:41.437 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 26 04:28:41 smithi098 podman[88308]: 2024-04-26 04:28:41.419252667 +0000 UTC m=+0.511819806 container remove 38f842279fa90180c4a5f8d6fdab6a608afdfdfead9dbba41a692553e447f3ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T04:28:41.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T04:28:41.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T04:28:41.494 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T04:28:41.494 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T04:28:41.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.1 2024-04-26T04:28:41.709 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:41 smithi098 systemd[1]: Stopping Ceph osd.1 for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:28:42.041 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:41 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1[44907]: 2024-04-26T04:28:41.805+0000 7fa2f2471700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:28:42.041 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:41 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1[44907]: 2024-04-26T04:28:41.805+0000 7fa2f2471700 -1 osd.1 21 *** Got signal Terminated *** 2024-04-26T04:28:42.042 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:41 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1[44907]: 2024-04-26T04:28:41.805+0000 7fa2f2471700 -1 osd.1 21 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T04:28:47.408 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:47 smithi098 podman[88398]: 2024-04-26 04:28:47.124291468 +0000 UTC m=+5.427362923 container died aecb356b08b18f81454c98d3ec35112bdb68c5f60a9215db7c3ecafd7cdd7513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, ceph=True) 2024-04-26T04:28:47.775 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:47 smithi098 podman[88398]: 2024-04-26 04:28:47.480957805 +0000 UTC m=+5.784029257 container cleanup aecb356b08b18f81454c98d3ec35112bdb68c5f60a9215db7c3ecafd7cdd7513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T04:28:47.775 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:47 smithi098 bash[88398]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1 2024-04-26T04:28:48.041 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:47 smithi098 podman[88427]: 2024-04-26 04:28:47.774097181 +0000 UTC m=+0.640614304 container remove aecb356b08b18f81454c98d3ec35112bdb68c5f60a9215db7c3ecafd7cdd7513 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, ceph=True) 2024-04-26T04:28:48.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.009701425 +0000 UTC m=+0.018195030 image pull b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4 2024-04-26T04:28:48.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.108408369 +0000 UTC m=+0.116902015 container create 2abdaea5f94efc59709a781c82b7a4b1f2ebc8380f723838cb6bb8b419d77e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.schema-version=1.0) 2024-04-26T04:28:48.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.319505227 +0000 UTC m=+0.327998814 container init 2abdaea5f94efc59709a781c82b7a4b1f2ebc8380f723838cb6bb8b419d77e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91) 2024-04-26T04:28:48.395 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.323133179 +0000 UTC m=+0.331626767 container start 2abdaea5f94efc59709a781c82b7a4b1f2ebc8380f723838cb6bb8b419d77e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T04:28:48.396 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.394480848 +0000 UTC m=+0.402974438 container attach 2abdaea5f94efc59709a781c82b7a4b1f2ebc8380f723838cb6bb8b419d77e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-26T04:28:48.791 INFO:journalctl@ceph.osd.1.smithi098.stdout:Apr 26 04:28:48 smithi098 podman[88521]: 2024-04-26 04:28:48.610956304 +0000 UTC m=+0.619449897 container died 2abdaea5f94efc59709a781c82b7a4b1f2ebc8380f723838cb6bb8b419d77e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T04:28:49.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T04:28:49.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T04:28:49.221 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T04:28:49.221 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T04:28:49.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.2 2024-04-26T04:28:49.511 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:49 smithi098 systemd[1]: Stopping Ceph osd.2 for b735c86e-0383-11ef-bc93-c7b262605968... 2024-04-26T04:28:49.791 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:49 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2[51394]: 2024-04-26T04:28:49.510+0000 7fad280ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T04:28:49.792 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:49 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2[51394]: 2024-04-26T04:28:49.510+0000 7fad280ad700 -1 osd.2 21 *** Got signal Terminated *** 2024-04-26T04:28:49.792 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:49 smithi098 ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2[51394]: 2024-04-26T04:28:49.510+0000 7fad280ad700 -1 osd.2 21 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T04:28:55.152 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:54 smithi098 podman[88634]: 2024-04-26 04:28:54.869044487 +0000 UTC m=+5.456849567 container died f62ef08a218dd3c40b4007982f59a7eea76c8660fa49c209477aebe2a38044d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-26T04:28:55.478 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:55 smithi098 podman[88634]: 2024-04-26 04:28:55.250948354 +0000 UTC m=+5.838753430 container cleanup f62ef08a218dd3c40b4007982f59a7eea76c8660fa49c209477aebe2a38044d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T04:28:55.478 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:55 smithi098 bash[88634]: ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2 2024-04-26T04:28:55.791 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:55 smithi098 podman[88651]: 2024-04-26 04:28:55.477076048 +0000 UTC m=+0.602647513 container remove f62ef08a218dd3c40b4007982f59a7eea76c8660fa49c209477aebe2a38044d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T04:28:56.062 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:55 smithi098 podman[88744]: 2024-04-26 04:28:55.701993525 +0000 UTC m=+0.018977827 image pull b2d50899180cb1370c09a2098e55a11930baef4ccad92b492a8f3682c3f965b3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4 2024-04-26T04:28:56.348 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:56 smithi098 podman[88744]: 2024-04-26 04:28:56.06137617 +0000 UTC m=+0.378360505 container create f613fc0b7c1d9743984f564df6fc566df9d9bd9b2f0635311976dce5c66fc28c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-26T04:28:56.348 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:56 smithi098 podman[88744]: 2024-04-26 04:28:56.273306231 +0000 UTC m=+0.590290520 container init f613fc0b7c1d9743984f564df6fc566df9d9bd9b2f0635311976dce5c66fc28c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T04:28:56.348 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:56 smithi098 podman[88744]: 2024-04-26 04:28:56.276621456 +0000 UTC m=+0.593605746 container start f613fc0b7c1d9743984f564df6fc566df9d9bd9b2f0635311976dce5c66fc28c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T04:28:56.348 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:56 smithi098 podman[88744]: 2024-04-26 04:28:56.347417554 +0000 UTC m=+0.664401859 container attach f613fc0b7c1d9743984f564df6fc566df9d9bd9b2f0635311976dce5c66fc28c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-26T04:28:56.791 INFO:journalctl@ceph.osd.2.smithi098.stdout:Apr 26 04:28:56 smithi098 podman[88744]: 2024-04-26 04:28:56.571323357 +0000 UTC m=+0.888307641 container died f613fc0b7c1d9743984f564df6fc566df9d9bd9b2f0635311976dce5c66fc28c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:5d90a071bff729ee9fb23d52aa874dbb8b26faeec6005f28fa39cc5f56e5c3a4, name=ceph-b735c86e-0383-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-yuri2-testing-2024-04-24-0914-squid-ddeaa91, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-26T04:28:57.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b735c86e-0383-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T04:28:57.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T04:28:57.180 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T04:28:57.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b735c86e-0383-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T04:28:57.374 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:28:58.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b735c86e-0383-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T04:28:59.070 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:28:59.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:28:59.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T04:28:59.426 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T04:28:59.461 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi098/crash 2024-04-26T04:28:59.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash -- . 2024-04-26T04:28:59.514 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T04:28:59.515 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-04-26T04:28:59.517 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi165/crash 2024-04-26T04:28:59.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash -- . 2024-04-26T04:28:59.551 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/b735c86e-0383-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T04:28:59.552 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-04-26T04:28:59.553 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T04:28:59.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-26T04:28:59.594 INFO:tasks.cephadm:Compressing logs... 2024-04-26T04:28:59.595 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T04:28:59.644 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T04:28:59.673 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T04:28:59.673 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-04-26T04:28:59.673 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T04:28:59.674 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 87.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T04:28:59.675 INFO:teuthology.orchestra.run.smithi165.stderr: 90.4% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T04:28:59.676 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-26T04:28:59.677 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m0.013s 2024-04-26T04:28:59.677 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m0.007s 2024-04-26T04:28:59.677 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.016s 2024-04-26T04:28:59.678 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.log 2024-04-26T04:28:59.679 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-04-26T04:28:59.679 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T04:28:59.679 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T04:28:59.679 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T04:28:59.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T04:28:59.684 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T04:28:59.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T04:28:59.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.1.log: /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T04:28:59.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mon.a.log: 89.2% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T04:28:59.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mgr.a.log 2024-04-26T04:28:59.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose --/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T04:28:59.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mgr.a.log: 81.0% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T04:28:59.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T04:28:59.687 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T04:28:59.691 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T04:28:59.728 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T04:28:59.746 INFO:teuthology.orchestra.run.smithi098.stderr: 90.5% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mgr.a.log.gz 2024-04-26T04:28:59.746 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T04:28:59.771 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T04:28:59.822 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/b735c86e-0383-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T04:28:59.825 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-26T04:28:59.825 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m0.156s 2024-04-26T04:28:59.825 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m0.426s 2024-04-26T04:28:59.825 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.038s 2024-04-26T04:28:59.825 INFO:tasks.cephadm:Archiving logs... 2024-04-26T04:28:59.826 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi098/log 2024-04-26T04:28:59.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T04:29:00.014 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi165/log 2024-04-26T04:29:00.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T04:29:00.045 INFO:tasks.cephadm:Removing cluster... 2024-04-26T04:29:00.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b735c86e-0383-11ef-bc93-c7b262605968 --force 2024-04-26T04:29:00.237 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:29:00.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b735c86e-0383-11ef-bc93-c7b262605968 --force 2024-04-26T04:29:00.679 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: b735c86e-0383-11ef-bc93-c7b262605968 2024-04-26T04:29:00.946 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T04:29:00.946 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T04:29:00.975 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T04:29:00.990 INFO:tasks.cephadm:Teardown complete 2024-04-26T04:29:00.991 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T04:29:01.002 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T04:29:01.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T04:29:01.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T04:29:01.104 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T04:29:01.114 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T04:29:01.114 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:29:01.122 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:29:01.138 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-26T04:29:01.143 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 +1289us[+1097us] +/- 47ms 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -1494us[-1494us] +/- 52ms 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -94us[ -94us] +/- 92ms 2024-04-26T04:29:01.144 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:29:01.147 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T04:29:01.148 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-26T04:29:01.148 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 +1259us[+1209us] +/- 47ms 2024-04-26T04:29:01.148 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 57 -1500us[-1550us] +/- 52ms 2024-04-26T04:29:01.148 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -110us[ -110us] +/- 92ms 2024-04-26T04:29:01.148 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T04:29:01.149 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T04:29:01.164 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T04:29:01.165 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T04:29:01.175 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T04:29:01.264 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T04:29:01.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-26T04:29:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:29:01.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-26T04:29:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:29:01.390 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T04:29:01.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T04:29:01.488 INFO:teuthology.task.internal:Duration was 1110.433105 seconds 2024-04-26T04:29:01.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T04:29:01.556 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T04:29:01.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T04:29:01.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T04:29:01.596 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T04:29:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T04:29:01.797 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T04:29:01.798 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-04-26T04:29:01.798 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T04:29:01.831 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-04-26T04:29:01.831 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T04:29:01.858 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T04:29:01.859 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T04:29:01.874 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T04:29:01.925 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T04:29:01.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T04:29:01.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T04:29:02.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T04:29:02.215 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T04:29:02.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T04:29:02.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T04:29:02.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T04:29:02.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:29:02.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:29:02.327 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-04-26T04:29:02.339 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-04-26T04:29:02.351 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:29:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:29:02.413 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:29:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:29:02.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T04:29:02.437 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T04:29:02.438 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi098 2024-04-26T04:29:02.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T04:29:02.503 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530/remote/smithi165 2024-04-26T04:29:02.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T04:29:02.537 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T04:29:02.538 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T04:29:02.541 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T04:29:02.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T04:29:02.597 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T04:29:02.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T04:29:02.610 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T04:29:02.610 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T04:29:02.613 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T04:29:02.638 INFO:teuthology.orchestra.run.smithi098.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 26 04:29 /home/ubuntu/cephtest 2024-04-26T04:29:02.638 INFO:teuthology.orchestra.run.smithi098.stdout: 525326 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 26 04:21 /home/ubuntu/cephtest/mnt.0 2024-04-26T04:29:02.638 INFO:teuthology.orchestra.run.smithi098.stdout: 525407 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 26 04:22 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T04:29:02.638 INFO:teuthology.orchestra.run.smithi098.stdout: 669544 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 04:22 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-26T04:29:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:29:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-26T04:29:02.640 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-26T04:29:02.646 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T04:29:02.658 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T04:29:02.670 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi098 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-26T04:29:02.670 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T04:29:02.676 INFO:teuthology.orchestra.run.smithi165.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 04:29 /home/ubuntu/cephtest 2024-04-26T04:29:02.702 DEBUG:teuthology.nuke:shortname: smithi098 2024-04-26T04:29:02.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T04:29:02.728 DEBUG:teuthology.nuke:shortname: smithi165 2024-04-26T04:29:02.729 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T04:29:02.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 04:04:56.609114', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA19QJu9tRoUX9UlbOntWO0tv02f2K/Fm2bg7NRZt1R0uJ8xt4JKE6c8XbnphlmwbhQ2/oPyZznlcxtcS1qmC2U='} 2024-04-26T04:29:02.757 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 04:04:56.610441', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKp4a3VjINqqXaKOYIbdLtD8d9Om8OH5J5nOOYvdQ2sGITObSENb1VqlE/B8l9oQ2LMFv7RTR8AP7UDFC2fvGos='} 2024-04-26T04:29:02.795 INFO:teuthology.orchestra.console.smithi098:Power off 2024-04-26T04:29:02.796 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T04:29:02.825 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-04-26T04:29:02.826 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:02.846 INFO:teuthology.orchestra.console.smithi165:Power off 2024-04-26T04:29:02.847 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T04:29:02.868 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-26T04:29:02.884 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-04-26T04:29:02.884 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:02.913 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-26T04:29:06.869 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:06.914 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:06.983 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-26T04:29:07.025 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-26T04:29:10.987 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:11.027 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:11.099 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-04-26T04:29:11.139 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-26T04:29:11.201 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-04-26T04:29:15.142 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T04:29:15.254 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-04-26T04:29:15.355 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-04-26T04:29:15.461 INFO:teuthology.run:Summary data: description: rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} duration: 1110.4331052303314 failure_reason: 'Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi098 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ddeaa91797339a19172c3036ff48cca58c12f448 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=96b4ef22a2f540d9b16de4ba220fc1ed status: fail success: false 2024-04-26T04:29:15.462 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T04:29:15.548 INFO:teuthology.run:FAIL